This is a HTML-ification of Workflow state (workflow.state) and
the associated files in the workflow output directory.
Below is the doc section of the workflow (.workflow file).
Produce, transform and plot fio-output ====================================== This workflow assumes that the 'prep.workflow' has completed. The comparison and plots consists of * bandwidth in KiB / seconds * latency is mean and in nano-seconds * IOPS are 'raw' / base-unit The above in a bar-plot at iodepth=1, and a line-plot as a function of iodepth. See Artifacts for the generated plots
If a workflow step (worklet / run) or test produced any artifacts, then they are listed here.
This is the configuration (.config file) with any
variable-placeholders such as {{ local.env.HOME }} filled out.
devices:
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme0n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme1n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme2n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme3n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme4n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme5n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme6n1
- driver_attachment: kernel
labels:
- dev
- bdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/nvme7n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng0n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng1n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng2n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng3n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng4n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng5n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng6n1
- driver_attachment: kernel
labels:
- dev
- cdev
- nvm
- scc
nsid: 1
spdk_conf: null
uri: /dev/ng7n1
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:41:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:42:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:43:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:44:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:45:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:46:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:47:00.0'
- driver_attachment: userspace
labels:
- dev
- pcie
- nvm
- scc
nsid: 1
spdk_conf: null
uri: '0000:48:00.0'
fio:
bin: /opt/fio/bin/fio
build:
prefix: /opt/fio
engines:
io_uring:
type: builtin
io_uring_cmd:
type: builtin
libaio:
type: builtin
spdk_bdev:
path: /root/git/spdk/build/fio/spdk_bdev
type: external_preload
spdk_nvme:
path: /root/git/spdk/build/fio/spdk_nvme
type: external_preload
xnvme:
path: /usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so
type: external_dynamic
repository:
path: /root/git/fio
remote: https://github.com/axboe/fio.git
tag: fio-3.32
liburing:
repository:
path: /root/git/liburing
remote: https://github.com/axboe/liburing.git
tag: liburing-2.2
os:
name: debian
version: bullseye
spdk:
build:
prefix: /opt/spdk
repository:
branch: for_bdev_xnvme_v6
path: /root/git/spdk
remote: https://github.com/safl/spdk.git
transport:
ssh:
hostname: fabio
port: 22
username: root
xnvme:
build:
type: debug
repository:
path: /root/git/xnvme
remote: https://github.com/OpenMPDK/xNVMe.git
tag: v0.6.0
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Run fio...
{'description': '\n Run fio...\n',
'extras': {'runlog': {'cmd_01': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.output'),
'state': {'begin': 1666743060.3786855,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8657588958740234,
'end': 1666743061.2444444,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.state')},
'cmd_02': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'171834.77 IOPS 83.90 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'171834.77 IOPS 83.90 MiB/s\n'
'[2022-10-26 04:09:12.283172] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:09:12.283242] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:09:12.283253] bdevperf '
'[2022-10-26 04:09:12.283263] '
'--no-shconf [2022-10-26 '
'04:09:12.283271] -l 0 [2022-10-26 '
'04:09:12.283280] --huge-unlink '
'[2022-10-26 04:09:12.283289] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:09:12.283296] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:09:12.283304] '
'--log-level=user1:6 [2022-10-26 '
'04:09:12.283312] --iova-mode=pa '
'[2022-10-26 04:09:12.283320] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:09:12.283328] '
'--match-allocations [2022-10-26 '
'04:09:12.283335] '
'--file-prefix=spdk_pid137152 '
'[2022-10-26 04:09:12.283344] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:09:12.421601] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:09:12.473509] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:09:12.474184] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.output'),
'state': {'begin': 1666743062.2779996,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.123723030090332,
'end': 1666743073.4017227,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.state')},
'cmd_03': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.output'),
'state': {'begin': 1666743073.4047213,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7753987312316895,
'end': 1666743074.18012,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.state')},
'cmd_04': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'172633.05 IOPS 84.29 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'172633.05 IOPS 84.29 MiB/s\n'
'[2022-10-26 04:09:25.202810] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:09:25.202878] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:09:25.202890] bdevperf '
'[2022-10-26 04:09:25.202898] '
'--no-shconf [2022-10-26 '
'04:09:25.202906] -l 0 [2022-10-26 '
'04:09:25.202914] --huge-unlink '
'[2022-10-26 04:09:25.202921] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:09:25.202930] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:09:25.202937] '
'--log-level=user1:6 [2022-10-26 '
'04:09:25.202944] --iova-mode=pa '
'[2022-10-26 04:09:25.202952] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:09:25.202959] '
'--match-allocations [2022-10-26 '
'04:09:25.202967] '
'--file-prefix=spdk_pid137177 '
'[2022-10-26 04:09:25.202974] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:09:25.341687] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:09:25.389529] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:09:25.390195] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.output'),
'state': {'begin': 1666743075.2015347,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.11318588256836,
'end': 1666743086.3147206,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.state')},
'cmd_05': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.output'),
'state': {'begin': 1666743086.3168955,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.782501220703125,
'end': 1666743087.0993967,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.state')},
'cmd_06': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'171552.75 IOPS 83.77 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'171552.75 IOPS 83.77 MiB/s\n'
'[2022-10-26 04:09:38.130730] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:09:38.130800] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:09:38.130811] bdevperf '
'[2022-10-26 04:09:38.130819] '
'--no-shconf [2022-10-26 '
'04:09:38.130828] -l 0 [2022-10-26 '
'04:09:38.130836] --huge-unlink '
'[2022-10-26 04:09:38.130845] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:09:38.130853] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:09:38.130860] '
'--log-level=user1:6 [2022-10-26 '
'04:09:38.130868] --iova-mode=pa '
'[2022-10-26 04:09:38.130876] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:09:38.130885] '
'--match-allocations [2022-10-26 '
'04:09:38.130893] '
'--file-prefix=spdk_pid137202 '
'[2022-10-26 04:09:38.130901] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:09:38.269317] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:09:38.321522] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:09:38.322192] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.output'),
'state': {'begin': 1666743088.124827,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.126126766204834,
'end': 1666743099.2509537,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.state')},
'cmd_07': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.output'),
'state': {'begin': 1666743099.2537508,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7822954654693604,
'end': 1666743100.0360463,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.state')},
'cmd_08': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'129651.27 IOPS 63.31 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'129651.27 IOPS 63.31 MiB/s\n'
'[2022-10-26 04:09:51.072421] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:09:51.072495] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:09:51.072506] bdevperf '
'[2022-10-26 04:09:51.072515] '
'--no-shconf [2022-10-26 '
'04:09:51.072524] -l 0 [2022-10-26 '
'04:09:51.072532] --huge-unlink '
'[2022-10-26 04:09:51.072541] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:09:51.072549] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:09:51.072557] '
'--log-level=user1:6 [2022-10-26 '
'04:09:51.072566] --iova-mode=pa '
'[2022-10-26 04:09:51.072574] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:09:51.072583] '
'--match-allocations [2022-10-26 '
'04:09:51.072591] '
'--file-prefix=spdk_pid137229 '
'[2022-10-26 04:09:51.072600] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:09:51.210930] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:09:51.257547] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:09:51.258212] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.output'),
'state': {'begin': 1666743101.0615962,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.104482889175415,
'end': 1666743112.166079,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.state')},
'cmd_09': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.output'),
'state': {'begin': 1666743112.16912,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7827787399291992,
'end': 1666743112.9518988,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.state')},
'cmd_10': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'129366.45 IOPS 63.17 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'129366.45 IOPS 63.17 MiB/s\n'
'[2022-10-26 04:10:03.980551] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:10:03.980620] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:10:03.980631] bdevperf '
'[2022-10-26 04:10:03.980640] '
'--no-shconf [2022-10-26 '
'04:10:03.980648] -l 0 [2022-10-26 '
'04:10:03.980655] --huge-unlink '
'[2022-10-26 04:10:03.980663] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:10:03.980671] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:10:03.980678] '
'--log-level=user1:6 [2022-10-26 '
'04:10:03.980686] --iova-mode=pa '
'[2022-10-26 04:10:03.980693] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:10:03.980700] '
'--match-allocations [2022-10-26 '
'04:10:03.980708] '
'--file-prefix=spdk_pid137254 '
'[2022-10-26 04:10:03.980716] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:10:04.119553] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:10:04.161522] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:10:04.162184] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.output'),
'state': {'begin': 1666743113.9712324,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.115413665771484,
'end': 1666743125.086646,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.state')},
'cmd_100': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'519280.46 IOPS 253.55 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'519280.46 IOPS 253.55 MiB/s\n'
'[2022-10-26 04:19:47.360269] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:19:47.360339] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:19:47.360350] bdevperf '
'[2022-10-26 04:19:47.360359] '
'--no-shconf [2022-10-26 '
'04:19:47.360367] -l 0 '
'[2022-10-26 04:19:47.360375] '
'--huge-unlink [2022-10-26 '
'04:19:47.360384] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:19:47.360392] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:19:47.360399] '
'--log-level=user1:6 [2022-10-26 '
'04:19:47.360407] --iova-mode=pa '
'[2022-10-26 04:19:47.360416] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:19:47.360423] '
'--match-allocations [2022-10-26 '
'04:19:47.360431] '
'--file-prefix=spdk_pid138396 '
'[2022-10-26 04:19:47.360440] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:19:47.498685] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:19:47.549512] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:19:47.550170] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.output'),
'state': {'begin': 1666743697.3514316,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.117814302444458,
'end': 1666743708.469246,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.state')},
'cmd_101': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.output'),
'state': {'begin': 1666743708.4720542,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7855734825134277,
'end': 1666743709.2576277,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.state')},
'cmd_102': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'520408.18 IOPS 254.11 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'520408.18 IOPS 254.11 MiB/s\n'
'[2022-10-26 04:20:00.275836] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:20:00.275907] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:20:00.275918] bdevperf '
'[2022-10-26 04:20:00.275926] '
'--no-shconf [2022-10-26 '
'04:20:00.275934] -l 0 '
'[2022-10-26 04:20:00.275942] '
'--huge-unlink [2022-10-26 '
'04:20:00.275949] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:20:00.275957] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:20:00.275964] '
'--log-level=user1:6 [2022-10-26 '
'04:20:00.275972] --iova-mode=pa '
'[2022-10-26 04:20:00.275979] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:20:00.275987] '
'--match-allocations [2022-10-26 '
'04:20:00.275994] '
'--file-prefix=spdk_pid138421 '
'[2022-10-26 04:20:00.276001] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:20:00.414472] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:20:00.453507] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:20:00.454165] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.output'),
'state': {'begin': 1666743710.2809129,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.092379808425903,
'end': 1666743721.3732927,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.state')},
'cmd_103': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.output'),
'state': {'begin': 1666743721.37594,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7914619445800781,
'end': 1666743722.167402,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.state')},
'cmd_104': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'534233.37 IOPS 260.86 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'534233.37 IOPS 260.86 MiB/s\n'
'[2022-10-26 04:20:13.190404] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:20:13.190474] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:20:13.190484] bdevperf '
'[2022-10-26 04:20:13.190494] '
'--no-shconf [2022-10-26 '
'04:20:13.190502] -l 0 '
'[2022-10-26 04:20:13.190509] '
'--huge-unlink [2022-10-26 '
'04:20:13.190517] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:20:13.190525] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:20:13.190532] '
'--log-level=user1:6 [2022-10-26 '
'04:20:13.190539] --iova-mode=pa '
'[2022-10-26 04:20:13.190546] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:20:13.190554] '
'--match-allocations [2022-10-26 '
'04:20:13.190561] '
'--file-prefix=spdk_pid138447 '
'[2022-10-26 04:20:13.190568] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:20:13.328990] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:20:13.369543] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:20:13.370218] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.output'),
'state': {'begin': 1666743723.1897388,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.102015495300293,
'end': 1666743734.2917542,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.state')},
'cmd_105': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.output'),
'state': {'begin': 1666743734.2947047,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7894554138183594,
'end': 1666743735.08416,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.state')},
'cmd_106': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'527043.53 IOPS 257.35 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'527043.53 IOPS 257.35 MiB/s\n'
'[2022-10-26 04:20:26.114969] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:20:26.115039] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:20:26.115050] bdevperf '
'[2022-10-26 04:20:26.115059] '
'--no-shconf [2022-10-26 '
'04:20:26.115067] -l 0 '
'[2022-10-26 04:20:26.115075] '
'--huge-unlink [2022-10-26 '
'04:20:26.115083] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:20:26.115090] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:20:26.115097] '
'--log-level=user1:6 [2022-10-26 '
'04:20:26.115105] --iova-mode=pa '
'[2022-10-26 04:20:26.115123] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:20:26.115131] '
'--match-allocations [2022-10-26 '
'04:20:26.115138] '
'--file-prefix=spdk_pid138472 '
'[2022-10-26 04:20:26.115146] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:20:26.253677] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:20:26.301556] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:20:26.302227] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.output'),
'state': {'begin': 1666743736.121251,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.103857517242432,
'end': 1666743747.2251086,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.state')},
'cmd_107': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.output'),
'state': {'begin': 1666743747.2278082,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.792818546295166,
'end': 1666743748.0206268,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.state')},
'cmd_108': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'530831.57 IOPS 259.20 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'530831.57 IOPS 259.20 MiB/s\n'
'[2022-10-26 04:20:39.084853] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:20:39.084921] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:20:39.084931] bdevperf '
'[2022-10-26 04:20:39.084940] '
'--no-shconf [2022-10-26 '
'04:20:39.084949] -l 0 '
'[2022-10-26 04:20:39.084957] '
'--huge-unlink [2022-10-26 '
'04:20:39.084966] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:20:39.084974] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:20:39.084981] '
'--log-level=user1:6 [2022-10-26 '
'04:20:39.084990] --iova-mode=pa '
'[2022-10-26 04:20:39.084998] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:20:39.085006] '
'--match-allocations [2022-10-26 '
'04:20:39.085015] '
'--file-prefix=spdk_pid138497 '
'[2022-10-26 04:20:39.085023] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:20:39.223403] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:20:39.269614] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:20:39.270283] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.output'),
'state': {'begin': 1666743749.092826,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.099281311035156,
'end': 1666743760.1921072,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.state')},
'cmd_109': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.output'),
'state': {'begin': 1666743760.1946323,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7804458141326904,
'end': 1666743760.975078,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.state')},
'cmd_11': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.output'),
'state': {'begin': 1666743125.0898678,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7840397357940674,
'end': 1666743125.8739076,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.state')},
'cmd_110': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'533336.85 IOPS 260.42 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'533336.85 IOPS 260.42 MiB/s\n'
'[2022-10-26 04:20:52.015515] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:20:52.015584] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:20:52.015595] bdevperf '
'[2022-10-26 04:20:52.015604] '
'--no-shconf [2022-10-26 '
'04:20:52.015612] -l 0 '
'[2022-10-26 04:20:52.015620] '
'--huge-unlink [2022-10-26 '
'04:20:52.015628] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:20:52.015636] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:20:52.015643] '
'--log-level=user1:6 [2022-10-26 '
'04:20:52.015650] --iova-mode=pa '
'[2022-10-26 04:20:52.015658] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:20:52.015665] '
'--match-allocations [2022-10-26 '
'04:20:52.015673] '
'--file-prefix=spdk_pid138522 '
'[2022-10-26 04:20:52.015680] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:20:52.154077] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:20:52.193585] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:20:52.194258] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.output'),
'state': {'begin': 1666743762.00147,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.181623220443726,
'end': 1666743773.1830933,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.state')},
'cmd_111': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.output'),
'state': {'begin': 1666743773.1863654,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7893142700195312,
'end': 1666743773.9756796,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.state')},
'cmd_112': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'536250.58 IOPS 261.84 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'536250.58 IOPS 261.84 MiB/s\n'
'[2022-10-26 04:21:05.002701] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:21:05.002771] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:21:05.002782] bdevperf '
'[2022-10-26 04:21:05.002791] '
'--no-shconf [2022-10-26 '
'04:21:05.002800] -l 0 '
'[2022-10-26 04:21:05.002808] '
'--huge-unlink [2022-10-26 '
'04:21:05.002816] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:21:05.002823] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:21:05.002831] '
'--log-level=user1:6 [2022-10-26 '
'04:21:05.002846] --iova-mode=pa '
'[2022-10-26 04:21:05.002854] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:21:05.002861] '
'--match-allocations [2022-10-26 '
'04:21:05.002870] '
'--file-prefix=spdk_pid138547 '
'[2022-10-26 04:21:05.002878] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:21:05.141323] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:21:05.185509] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:21:05.186190] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.output'),
'state': {'begin': 1666743775.006602,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.164662837982178,
'end': 1666743786.171265,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.state')},
'cmd_113': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.output'),
'state': {'begin': 1666743786.1741786,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7843453884124756,
'end': 1666743786.958524,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.state')},
'cmd_114': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'534521.72 IOPS 261.00 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'534521.72 IOPS 261.00 MiB/s\n'
'[2022-10-26 04:21:17.969960] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:21:17.970030] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:21:17.970041] bdevperf '
'[2022-10-26 04:21:17.970050] '
'--no-shconf [2022-10-26 '
'04:21:17.970058] -l 0 '
'[2022-10-26 04:21:17.970066] '
'--huge-unlink [2022-10-26 '
'04:21:17.970074] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:21:17.970081] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:21:17.970088] '
'--log-level=user1:6 [2022-10-26 '
'04:21:17.970096] --iova-mode=pa '
'[2022-10-26 04:21:17.970103] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:21:17.970111] '
'--match-allocations [2022-10-26 '
'04:21:17.970118] '
'--file-prefix=spdk_pid138572 '
'[2022-10-26 04:21:17.970126] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:21:18.108770] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:21:18.149565] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:21:18.150241] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.output'),
'state': {'begin': 1666743787.9747603,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.15734577178955,
'end': 1666743799.132106,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.state')},
'cmd_115': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.output'),
'state': {'begin': 1666743799.1346714,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7830836772918701,
'end': 1666743799.9177551,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.state')},
'cmd_116': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'510627.58 IOPS 249.33 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'510627.58 IOPS 249.33 MiB/s\n'
'[2022-10-26 04:21:30.943091] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:21:30.943175] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:21:30.943186] bdevperf '
'[2022-10-26 04:21:30.943195] '
'--no-shconf [2022-10-26 '
'04:21:30.943204] -l 0 '
'[2022-10-26 04:21:30.943211] '
'--huge-unlink [2022-10-26 '
'04:21:30.943220] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:21:30.943227] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:21:30.943235] '
'--log-level=user1:6 [2022-10-26 '
'04:21:30.943243] --iova-mode=pa '
'[2022-10-26 04:21:30.943251] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:21:30.943259] '
'--match-allocations [2022-10-26 '
'04:21:30.943267] '
'--file-prefix=spdk_pid138597 '
'[2022-10-26 04:21:30.943275] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:21:31.081602] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:21:31.121496] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:21:31.122161] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.output'),
'state': {'begin': 1666743800.9505537,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.176557779312134,
'end': 1666743812.1271114,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.state')},
'cmd_117': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.output'),
'state': {'begin': 1666743812.1298392,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7924110889434814,
'end': 1666743812.9222503,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.state')},
'cmd_118': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'507405.68 IOPS 247.76 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'507405.68 IOPS 247.76 MiB/s\n'
'[2022-10-26 04:21:43.951587] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:21:43.951660] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:21:43.951671] bdevperf '
'[2022-10-26 04:21:43.951680] '
'--no-shconf [2022-10-26 '
'04:21:43.951688] -l 0 '
'[2022-10-26 04:21:43.951696] '
'--huge-unlink [2022-10-26 '
'04:21:43.951704] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:21:43.951712] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:21:43.951719] '
'--log-level=user1:6 [2022-10-26 '
'04:21:43.951726] --iova-mode=pa '
'[2022-10-26 04:21:43.951734] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:21:43.951741] '
'--match-allocations [2022-10-26 '
'04:21:43.951749] '
'--file-prefix=spdk_pid138623 '
'[2022-10-26 04:21:43.951756] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:21:44.090207] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:21:44.129577] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:21:44.130245] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.output'),
'state': {'begin': 1666743813.9570987,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.194416761398315,
'end': 1666743825.1515155,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.state')},
'cmd_119': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.output'),
'state': {'begin': 1666743825.1540108,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7881972789764404,
'end': 1666743825.942208,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.state')},
'cmd_12': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'129066.85 IOPS 63.02 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'129066.85 IOPS 63.02 MiB/s\n'
'[2022-10-26 04:10:16.900959] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:10:16.901031] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:10:16.901042] bdevperf '
'[2022-10-26 04:10:16.901050] '
'--no-shconf [2022-10-26 '
'04:10:16.901058] -l 0 [2022-10-26 '
'04:10:16.901066] --huge-unlink '
'[2022-10-26 04:10:16.901074] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:10:16.901081] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:10:16.901088] '
'--log-level=user1:6 [2022-10-26 '
'04:10:16.901096] --iova-mode=pa '
'[2022-10-26 04:10:16.901104] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:10:16.901111] '
'--match-allocations [2022-10-26 '
'04:10:16.901118] '
'--file-prefix=spdk_pid137279 '
'[2022-10-26 04:10:16.901126] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:10:17.039698] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:10:17.081578] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:10:17.082253] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.output'),
'state': {'begin': 1666743126.8989022,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.096217393875122,
'end': 1666743137.9951196,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.state')},
'cmd_120': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'510139.78 IOPS 249.09 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'510139.78 IOPS 249.09 MiB/s\n'
'[2022-10-26 04:21:56.963424] '
'Starting SPDK v23.01-pre git '
'sha1 fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:21:56.963490] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:21:56.963502] bdevperf '
'[2022-10-26 04:21:56.963511] '
'--no-shconf [2022-10-26 '
'04:21:56.963520] -l 0 '
'[2022-10-26 04:21:56.963527] '
'--huge-unlink [2022-10-26 '
'04:21:56.963536] '
'--log-level=lib.eal:6 '
'[2022-10-26 04:21:56.963543] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:21:56.963551] '
'--log-level=user1:6 [2022-10-26 '
'04:21:56.963559] --iova-mode=pa '
'[2022-10-26 04:21:56.963567] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:21:56.963575] '
'--match-allocations [2022-10-26 '
'04:21:56.963583] '
'--file-prefix=spdk_pid138648 '
'[2022-10-26 04:21:56.963590] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:21:57.102363] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:21:57.145569] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on '
'core 0\n'
'[2022-10-26 04:21:57.146250] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.output'),
'state': {'begin': 1666743826.971447,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 8 -o 512 -w randread '
"-t 10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.168527841567993,
'end': 1666743838.1399748,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.state')},
'cmd_13': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.output'),
'state': {'begin': 1666743138.0030165,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7842626571655273,
'end': 1666743138.7872791,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.state')},
'cmd_14': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'147294.48 IOPS 71.92 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'147294.48 IOPS 71.92 MiB/s\n'
'[2022-10-26 04:10:29.810550] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:10:29.810618] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:10:29.810628] bdevperf '
'[2022-10-26 04:10:29.810637] '
'--no-shconf [2022-10-26 '
'04:10:29.810646] -l 0 [2022-10-26 '
'04:10:29.810653] --huge-unlink '
'[2022-10-26 04:10:29.810662] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:10:29.810670] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:10:29.810677] '
'--log-level=user1:6 [2022-10-26 '
'04:10:29.810685] --iova-mode=pa '
'[2022-10-26 04:10:29.810694] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:10:29.810701] '
'--match-allocations [2022-10-26 '
'04:10:29.810710] '
'--file-prefix=spdk_pid137304 '
'[2022-10-26 04:10:29.810718] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:10:29.949133] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:10:29.993590] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:10:29.994247] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.output'),
'state': {'begin': 1666743139.8103492,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.096240758895874,
'end': 1666743150.90659,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.state')},
'cmd_15': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.output'),
'state': {'begin': 1666743150.9092145,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7844667434692383,
'end': 1666743151.6936812,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.state')},
'cmd_16': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'147811.38 IOPS 72.17 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'147811.38 IOPS 72.17 MiB/s\n'
'[2022-10-26 04:10:42.746033] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:10:42.746103] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:10:42.746113] bdevperf '
'[2022-10-26 04:10:42.746122] '
'--no-shconf [2022-10-26 '
'04:10:42.746130] -l 0 [2022-10-26 '
'04:10:42.746139] --huge-unlink '
'[2022-10-26 04:10:42.746146] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:10:42.746154] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:10:42.746161] '
'--log-level=user1:6 [2022-10-26 '
'04:10:42.746169] --iova-mode=pa '
'[2022-10-26 04:10:42.746176] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:10:42.746183] '
'--match-allocations [2022-10-26 '
'04:10:42.746191] '
'--file-prefix=spdk_pid137329 '
'[2022-10-26 04:10:42.746198] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:10:42.884755] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:10:42.934563] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:10:42.935300] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.output'),
'state': {'begin': 1666743152.73093,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.12259316444397,
'end': 1666743163.8535233,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.state')},
'cmd_17': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.output'),
'state': {'begin': 1666743163.8573275,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7823679447174072,
'end': 1666743164.6396954,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.state')},
'cmd_18': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'147836.09 IOPS 72.19 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'147836.09 IOPS 72.19 MiB/s\n'
'[2022-10-26 04:10:55.660577] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:10:55.660666] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:10:55.660678] bdevperf '
'[2022-10-26 04:10:55.660688] '
'--no-shconf [2022-10-26 '
'04:10:55.660697] -l 0 [2022-10-26 '
'04:10:55.660706] --huge-unlink '
'[2022-10-26 04:10:55.660716] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:10:55.660724] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:10:55.660733] '
'--log-level=user1:6 [2022-10-26 '
'04:10:55.660742] --iova-mode=pa '
'[2022-10-26 04:10:55.660750] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:10:55.660759] '
'--match-allocations [2022-10-26 '
'04:10:55.660768] '
'--file-prefix=spdk_pid137354 '
'[2022-10-26 04:10:55.660776] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:10:55.802085] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:10:55.849530] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:10:55.850223] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.output'),
'state': {'begin': 1666743165.662862,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.099805116653442,
'end': 1666743176.7626672,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.state')},
'cmd_19': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.output'),
'state': {'begin': 1666743176.7660782,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7962827682495117,
'end': 1666743177.562361,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.state')},
'cmd_20': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'145829.56 IOPS 71.21 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'145829.56 IOPS 71.21 MiB/s\n'
'[2022-10-26 04:11:08.598776] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:11:08.598845] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:11:08.598856] bdevperf '
'[2022-10-26 04:11:08.598865] '
'--no-shconf [2022-10-26 '
'04:11:08.598874] -l 0 [2022-10-26 '
'04:11:08.598882] --huge-unlink '
'[2022-10-26 04:11:08.598891] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:11:08.598898] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:11:08.598905] '
'--log-level=user1:6 [2022-10-26 '
'04:11:08.598915] --iova-mode=pa '
'[2022-10-26 04:11:08.598923] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:11:08.598931] '
'--match-allocations [2022-10-26 '
'04:11:08.598939] '
'--file-prefix=spdk_pid137380 '
'[2022-10-26 04:11:08.598947] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:11:08.737667] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:11:08.785519] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:11:08.786203] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.output'),
'state': {'begin': 1666743178.5923176,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.168691635131836,
'end': 1666743189.7610092,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.state')},
'cmd_21': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.output'),
'state': {'begin': 1666743189.7641158,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7852909564971924,
'end': 1666743190.5494068,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.state')},
'cmd_22': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'157191.61 IOPS 76.75 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'157191.61 IOPS 76.75 MiB/s\n'
'[2022-10-26 04:11:21.593539] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:11:21.593610] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:11:21.593621] bdevperf '
'[2022-10-26 04:11:21.593629] '
'--no-shconf [2022-10-26 '
'04:11:21.593638] -l 0 [2022-10-26 '
'04:11:21.593646] --huge-unlink '
'[2022-10-26 04:11:21.593653] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:11:21.593661] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:11:21.593669] '
'--log-level=user1:6 [2022-10-26 '
'04:11:21.593677] --iova-mode=pa '
'[2022-10-26 04:11:21.593685] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:11:21.593693] '
'--match-allocations [2022-10-26 '
'04:11:21.593701] '
'--file-prefix=spdk_pid137406 '
'[2022-10-26 04:11:21.593709] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:11:21.732218] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:11:21.777567] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:11:21.778256] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.output'),
'state': {'begin': 1666743191.5794482,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.181533575057983,
'end': 1666743202.7609818,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.state')},
'cmd_23': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.output'),
'state': {'begin': 1666743202.7638466,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.796776533126831,
'end': 1666743203.5606232,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.state')},
'cmd_24': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'145943.30 IOPS 71.26 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'145943.30 IOPS 71.26 MiB/s\n'
'[2022-10-26 04:11:34.584729] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:11:34.584800] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:11:34.584811] bdevperf '
'[2022-10-26 04:11:34.584820] '
'--no-shconf [2022-10-26 '
'04:11:34.584829] -l 0 [2022-10-26 '
'04:11:34.584837] --huge-unlink '
'[2022-10-26 04:11:34.584846] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:11:34.584854] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:11:34.584861] '
'--log-level=user1:6 [2022-10-26 '
'04:11:34.584869] --iova-mode=pa '
'[2022-10-26 04:11:34.584877] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:11:34.584885] '
'--match-allocations [2022-10-26 '
'04:11:34.584893] '
'--file-prefix=spdk_pid137431 '
'[2022-10-26 04:11:34.584902] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:11:34.723185] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:11:34.765492] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:11:34.766168] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.output'),
'state': {'begin': 1666743204.5806067,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.164729356765747,
'end': 1666743215.745336,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.state')},
'cmd_25': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.output'),
'state': {'begin': 1666743215.748288,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.791050910949707,
'end': 1666743216.5393388,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.state')},
'cmd_26': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'151211.44 IOPS 73.83 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'151211.44 IOPS 73.83 MiB/s\n'
'[2022-10-26 04:11:47.573241] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:11:47.573309] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:11:47.573320] bdevperf '
'[2022-10-26 04:11:47.573329] '
'--no-shconf [2022-10-26 '
'04:11:47.573338] -l 0 [2022-10-26 '
'04:11:47.573346] --huge-unlink '
'[2022-10-26 04:11:47.573355] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:11:47.573363] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:11:47.573389] '
'--log-level=user1:6 [2022-10-26 '
'04:11:47.573398] --iova-mode=pa '
'[2022-10-26 04:11:47.573406] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:11:47.573414] '
'--match-allocations [2022-10-26 '
'04:11:47.573426] '
'--file-prefix=spdk_pid137456 '
'[2022-10-26 04:11:47.573434] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:11:47.711808] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:11:47.753532] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:11:47.754194] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.output'),
'state': {'begin': 1666743217.5643964,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.17328691482544,
'end': 1666743228.7376833,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.state')},
'cmd_27': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.output'),
'state': {'begin': 1666743228.7405007,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8015680313110352,
'end': 1666743229.5420687,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.state')},
'cmd_28': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'139449.94 IOPS 68.09 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'139449.94 IOPS 68.09 MiB/s\n'
'[2022-10-26 04:12:00.586747] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:12:00.586818] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:12:00.586829] bdevperf '
'[2022-10-26 04:12:00.586837] '
'--no-shconf [2022-10-26 '
'04:12:00.586845] -l 0 [2022-10-26 '
'04:12:00.586853] --huge-unlink '
'[2022-10-26 04:12:00.586860] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:12:00.586868] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:12:00.586875] '
'--log-level=user1:6 [2022-10-26 '
'04:12:00.586883] --iova-mode=pa '
'[2022-10-26 04:12:00.586890] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:12:00.586898] '
'--match-allocations [2022-10-26 '
'04:12:00.586905] '
'--file-prefix=spdk_pid137481 '
'[2022-10-26 04:12:00.586912] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:12:00.725651] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:12:00.765524] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:12:00.766197] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.output'),
'state': {'begin': 1666743230.5813017,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.189642906188965,
'end': 1666743241.7709446,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.state')},
'cmd_29': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.output'),
'state': {'begin': 1666743241.7739084,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7814905643463135,
'end': 1666743242.555399,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.state')},
'cmd_30': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'139549.74 IOPS 68.14 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'139549.74 IOPS 68.14 MiB/s\n'
'[2022-10-26 04:12:13.587573] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:12:13.587648] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:12:13.587659] bdevperf '
'[2022-10-26 04:12:13.587669] '
'--no-shconf [2022-10-26 '
'04:12:13.587678] -l 0 [2022-10-26 '
'04:12:13.587686] --huge-unlink '
'[2022-10-26 04:12:13.587695] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:12:13.587703] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:12:13.587711] '
'--log-level=user1:6 [2022-10-26 '
'04:12:13.587720] --iova-mode=pa '
'[2022-10-26 04:12:13.587728] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:12:13.587736] '
'--match-allocations [2022-10-26 '
'04:12:13.587744] '
'--file-prefix=spdk_pid137506 '
'[2022-10-26 04:12:13.587752] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:12:13.726489] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:12:13.773484] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:12:13.774158] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.output'),
'state': {'begin': 1666743243.586331,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 1 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.183897733688354,
'end': 1666743254.7702286,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.state')},
'cmd_31': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.output'),
'state': {'begin': 1666743254.7777295,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7828829288482666,
'end': 1666743255.5606124,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.state')},
'cmd_32': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'262928.44 IOPS 128.38 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'262928.44 IOPS 128.38 MiB/s\n'
'[2022-10-26 04:12:26.587184] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:12:26.587258] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:12:26.587270] bdevperf '
'[2022-10-26 04:12:26.587278] '
'--no-shconf [2022-10-26 '
'04:12:26.587287] -l 0 [2022-10-26 '
'04:12:26.587295] --huge-unlink '
'[2022-10-26 04:12:26.587304] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:12:26.587312] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:12:26.587319] '
'--log-level=user1:6 [2022-10-26 '
'04:12:26.587328] --iova-mode=pa '
'[2022-10-26 04:12:26.587337] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:12:26.587344] '
'--match-allocations [2022-10-26 '
'04:12:26.587352] '
'--file-prefix=spdk_pid137531 '
'[2022-10-26 04:12:26.587360] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:12:26.725751] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:12:26.765496] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:12:26.766167] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.output'),
'state': {'begin': 1666743256.5828512,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.110819339752197,
'end': 1666743267.6936705,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.state')},
'cmd_33': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.output'),
'state': {'begin': 1666743267.696667,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.8754153251647949,
'end': 1666743268.5720823,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.state')},
'cmd_34': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'242424.58 IOPS 118.37 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'242424.58 IOPS 118.37 MiB/s\n'
'[2022-10-26 04:12:39.856534] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:12:39.856602] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:12:39.856613] bdevperf '
'[2022-10-26 04:12:39.856621] '
'--no-shconf [2022-10-26 '
'04:12:39.856629] -l 0 [2022-10-26 '
'04:12:39.856637] --huge-unlink '
'[2022-10-26 04:12:39.856644] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:12:39.856652] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:12:39.856659] '
'--log-level=user1:6 [2022-10-26 '
'04:12:39.856667] --iova-mode=pa '
'[2022-10-26 04:12:39.856674] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:12:39.856681] '
'--match-allocations [2022-10-26 '
'04:12:39.856688] '
'--file-prefix=spdk_pid137556 '
'[2022-10-26 04:12:39.856696] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:12:39.994898] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:12:40.041502] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:12:40.042162] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.output'),
'state': {'begin': 1666743269.7216797,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.253293752670288,
'end': 1666743280.9749734,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.state')},
'cmd_35': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.output'),
'state': {'begin': 1666743280.9778373,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7800905704498291,
'end': 1666743281.757928,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.state')},
'cmd_36': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'263549.12 IOPS 128.69 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'263549.12 IOPS 128.69 MiB/s\n'
'[2022-10-26 04:12:52.784402] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:12:52.784475] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:12:52.784486] bdevperf '
'[2022-10-26 04:12:52.784494] '
'--no-shconf [2022-10-26 '
'04:12:52.784503] -l 0 [2022-10-26 '
'04:12:52.784511] --huge-unlink '
'[2022-10-26 04:12:52.784520] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:12:52.784527] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:12:52.784536] '
'--log-level=user1:6 [2022-10-26 '
'04:12:52.784544] --iova-mode=pa '
'[2022-10-26 04:12:52.784552] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:12:52.784560] '
'--match-allocations [2022-10-26 '
'04:12:52.784568] '
'--file-prefix=spdk_pid137581 '
'[2022-10-26 04:12:52.784576] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:12:52.923433] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:12:52.969615] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:12:52.970299] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.output'),
'state': {'begin': 1666743282.7862253,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.112562894821167,
'end': 1666743293.8987882,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.state')},
'cmd_37': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.output'),
'state': {'begin': 1666743293.9017942,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7872776985168457,
'end': 1666743294.689072,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.state')},
'cmd_38': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'228150.59 IOPS 111.40 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'228150.59 IOPS 111.40 MiB/s\n'
'[2022-10-26 04:13:05.723004] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:13:05.723072] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:13:05.723082] bdevperf '
'[2022-10-26 04:13:05.723091] '
'--no-shconf [2022-10-26 '
'04:13:05.723100] -l 0 [2022-10-26 '
'04:13:05.723108] --huge-unlink '
'[2022-10-26 04:13:05.723127] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:13:05.723136] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:13:05.723143] '
'--log-level=user1:6 [2022-10-26 '
'04:13:05.723151] --iova-mode=pa '
'[2022-10-26 04:13:05.723159] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:13:05.723167] '
'--match-allocations [2022-10-26 '
'04:13:05.723176] '
'--file-prefix=spdk_pid137607 '
'[2022-10-26 04:13:05.723184] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:13:05.862734] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:13:05.917790] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:13:05.918471] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.output'),
'state': {'begin': 1666743295.7155757,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.114653825759888,
'end': 1666743306.8302295,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.state')},
'cmd_39': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.output'),
'state': {'begin': 1666743306.8331847,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.784264087677002,
'end': 1666743307.6174488,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.state')},
'cmd_40': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'230398.16 IOPS 112.50 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'230398.16 IOPS 112.50 MiB/s\n'
'[2022-10-26 04:13:18.651302] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:13:18.651371] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:13:18.651381] bdevperf '
'[2022-10-26 04:13:18.651390] '
'--no-shconf [2022-10-26 '
'04:13:18.651399] -l 0 [2022-10-26 '
'04:13:18.651407] --huge-unlink '
'[2022-10-26 04:13:18.651416] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:13:18.651423] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:13:18.651431] '
'--log-level=user1:6 [2022-10-26 '
'04:13:18.651440] --iova-mode=pa '
'[2022-10-26 04:13:18.651448] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:13:18.651456] '
'--match-allocations [2022-10-26 '
'04:13:18.651464] '
'--file-prefix=spdk_pid137632 '
'[2022-10-26 04:13:18.651472] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:13:18.790232] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:13:18.829548] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:13:18.830238] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.output'),
'state': {'begin': 1666743308.6497045,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.093471765518188,
'end': 1666743319.7431762,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.state')},
'cmd_41': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.output'),
'state': {'begin': 1666743319.7462144,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7990851402282715,
'end': 1666743320.5452995,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.state')},
'cmd_42': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'229316.75 IOPS 111.97 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'229316.75 IOPS 111.97 MiB/s\n'
'[2022-10-26 04:13:31.605298] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:13:31.605366] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:13:31.605377] bdevperf '
'[2022-10-26 04:13:31.605386] '
'--no-shconf [2022-10-26 '
'04:13:31.605394] -l 0 [2022-10-26 '
'04:13:31.605402] --huge-unlink '
'[2022-10-26 04:13:31.605411] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:13:31.605419] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:13:31.605426] '
'--log-level=user1:6 [2022-10-26 '
'04:13:31.605434] --iova-mode=pa '
'[2022-10-26 04:13:31.605441] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:13:31.605449] '
'--match-allocations [2022-10-26 '
'04:13:31.605456] '
'--file-prefix=spdk_pid137657 '
'[2022-10-26 04:13:31.605464] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:13:31.743848] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:13:31.793535] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:13:31.794215] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.output'),
'state': {'begin': 1666743321.5992064,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.111653804779053,
'end': 1666743332.7108603,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.state')},
'cmd_43': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.output'),
'state': {'begin': 1666743332.713505,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7880523204803467,
'end': 1666743333.5015574,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.state')},
'cmd_44': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'291240.87 IOPS 142.21 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'291240.87 IOPS 142.21 MiB/s\n'
'[2022-10-26 04:13:44.534674] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:13:44.534744] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:13:44.534755] bdevperf '
'[2022-10-26 04:13:44.534764] '
'--no-shconf [2022-10-26 '
'04:13:44.534773] -l 0 [2022-10-26 '
'04:13:44.534780] --huge-unlink '
'[2022-10-26 04:13:44.534789] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:13:44.534797] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:13:44.534805] '
'--log-level=user1:6 [2022-10-26 '
'04:13:44.534819] --iova-mode=pa '
'[2022-10-26 04:13:44.534827] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:13:44.534835] '
'--match-allocations [2022-10-26 '
'04:13:44.534842] '
'--file-prefix=spdk_pid137682 '
'[2022-10-26 04:13:44.534850] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:13:44.673093] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:13:44.725564] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:13:44.726241] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.output'),
'state': {'begin': 1666743334.5314794,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.119513273239136,
'end': 1666743345.6509926,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.state')},
'cmd_45': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.output'),
'state': {'begin': 1666743345.6540956,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7841320037841797,
'end': 1666743346.4382277,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.state')},
'cmd_46': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'290441.54 IOPS 141.82 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'290441.54 IOPS 141.82 MiB/s\n'
'[2022-10-26 04:13:57.478245] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:13:57.478314] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:13:57.478324] bdevperf '
'[2022-10-26 04:13:57.478333] '
'--no-shconf [2022-10-26 '
'04:13:57.478341] -l 0 [2022-10-26 '
'04:13:57.478349] --huge-unlink '
'[2022-10-26 04:13:57.478357] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:13:57.478365] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:13:57.478372] '
'--log-level=user1:6 [2022-10-26 '
'04:13:57.478380] --iova-mode=pa '
'[2022-10-26 04:13:57.478387] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:13:57.478395] '
'--match-allocations [2022-10-26 '
'04:13:57.478402] '
'--file-prefix=spdk_pid137708 '
'[2022-10-26 04:13:57.478409] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:13:57.617477] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:13:57.661885] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:13:57.662595] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.output'),
'state': {'begin': 1666743347.4800632,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.106460094451904,
'end': 1666743358.5865233,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.state')},
'cmd_47': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.output'),
'state': {'begin': 1666743358.5895798,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.786015510559082,
'end': 1666743359.3755953,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.state')},
'cmd_48': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'290486.27 IOPS 141.84 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'290486.27 IOPS 141.84 MiB/s\n'
'[2022-10-26 04:14:10.393432] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:14:10.393501] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:14:10.393511] bdevperf '
'[2022-10-26 04:14:10.393520] '
'--no-shconf [2022-10-26 '
'04:14:10.393528] -l 0 [2022-10-26 '
'04:14:10.393536] --huge-unlink '
'[2022-10-26 04:14:10.393544] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:14:10.393552] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:14:10.393559] '
'--log-level=user1:6 [2022-10-26 '
'04:14:10.393567] --iova-mode=pa '
'[2022-10-26 04:14:10.393575] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:14:10.393583] '
'--match-allocations [2022-10-26 '
'04:14:10.393590] '
'--file-prefix=spdk_pid137735 '
'[2022-10-26 04:14:10.393598] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:14:10.531814] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:14:10.581554] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:14:10.582236] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.output'),
'state': {'begin': 1666743360.389052,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.11704397201538,
'end': 1666743371.506096,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.state')},
'cmd_49': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.output'),
'state': {'begin': 1666743371.509128,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7821986675262451,
'end': 1666743372.2913268,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.state')},
'cmd_50': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'285675.37 IOPS 139.49 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'285675.37 IOPS 139.49 MiB/s\n'
'[2022-10-26 04:14:23.317220] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:14:23.317292] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:14:23.317303] bdevperf '
'[2022-10-26 04:14:23.317312] '
'--no-shconf [2022-10-26 '
'04:14:23.317320] -l 0 [2022-10-26 '
'04:14:23.317329] --huge-unlink '
'[2022-10-26 04:14:23.317338] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:14:23.317345] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:14:23.317352] '
'--log-level=user1:6 [2022-10-26 '
'04:14:23.317360] --iova-mode=pa '
'[2022-10-26 04:14:23.317367] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:14:23.317375] '
'--match-allocations [2022-10-26 '
'04:14:23.317383] '
'--file-prefix=spdk_pid137760 '
'[2022-10-26 04:14:23.317391] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:14:23.456191] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:14:23.509543] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:14:23.510240] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.output'),
'state': {'begin': 1666743373.3197203,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.168123722076416,
'end': 1666743384.487844,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.state')},
'cmd_51': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.output'),
'state': {'begin': 1666743384.4903607,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7734940052032471,
'end': 1666743385.2638547,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.state')},
'cmd_52': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'285787.73 IOPS 139.54 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'285787.73 IOPS 139.54 MiB/s\n'
'[2022-10-26 04:14:36.284758] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:14:36.284827] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:14:36.284838] bdevperf '
'[2022-10-26 04:14:36.284847] '
'--no-shconf [2022-10-26 '
'04:14:36.284855] -l 0 [2022-10-26 '
'04:14:36.284864] --huge-unlink '
'[2022-10-26 04:14:36.284873] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:14:36.284880] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:14:36.284888] '
'--log-level=user1:6 [2022-10-26 '
'04:14:36.284896] --iova-mode=pa '
'[2022-10-26 04:14:36.284905] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:14:36.284913] '
'--match-allocations [2022-10-26 '
'04:14:36.284921] '
'--file-prefix=spdk_pid137786 '
'[2022-10-26 04:14:36.284929] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:14:36.423316] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:14:36.465562] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:14:36.466223] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.output'),
'state': {'begin': 1666743386.2882006,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.159849882125854,
'end': 1666743397.4480505,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.state')},
'cmd_53': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.output'),
'state': {'begin': 1666743397.4523597,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7845513820648193,
'end': 1666743398.236911,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.state')},
'cmd_54': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'283717.83 IOPS 138.53 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'283717.83 IOPS 138.53 MiB/s\n'
'[2022-10-26 04:14:49.262571] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:14:49.262640] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:14:49.262651] bdevperf '
'[2022-10-26 04:14:49.262661] '
'--no-shconf [2022-10-26 '
'04:14:49.262669] -l 0 [2022-10-26 '
'04:14:49.262676] --huge-unlink '
'[2022-10-26 04:14:49.262684] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:14:49.262692] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:14:49.262699] '
'--log-level=user1:6 [2022-10-26 '
'04:14:49.262706] --iova-mode=pa '
'[2022-10-26 04:14:49.262714] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:14:49.262722] '
'--match-allocations [2022-10-26 '
'04:14:49.262729] '
'--file-prefix=spdk_pid137811 '
'[2022-10-26 04:14:49.262737] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:14:49.401169] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:14:49.449515] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:14:49.450200] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.output'),
'state': {'begin': 1666743399.2636812,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.170012474060059,
'end': 1666743410.4336936,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.state')},
'cmd_55': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.output'),
'state': {'begin': 1666743410.4362636,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7805120944976807,
'end': 1666743411.2167757,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.state')},
'cmd_56': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'273786.89 IOPS 133.69 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'273786.89 IOPS 133.69 MiB/s\n'
'[2022-10-26 04:15:02.256385] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:15:02.256453] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:15:02.256464] bdevperf '
'[2022-10-26 04:15:02.256473] '
'--no-shconf [2022-10-26 '
'04:15:02.256481] -l 0 [2022-10-26 '
'04:15:02.256489] --huge-unlink '
'[2022-10-26 04:15:02.256498] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:15:02.256506] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:15:02.256514] '
'--log-level=user1:6 [2022-10-26 '
'04:15:02.256522] --iova-mode=pa '
'[2022-10-26 04:15:02.256531] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:15:02.256539] '
'--match-allocations [2022-10-26 '
'04:15:02.256548] '
'--file-prefix=spdk_pid137838 '
'[2022-10-26 04:15:02.256556] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:15:02.395084] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:15:02.453546] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:15:02.454235] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.output'),
'state': {'begin': 1666743412.2478497,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.20577597618103,
'end': 1666743423.4536257,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.state')},
'cmd_57': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.output'),
'state': {'begin': 1666743423.456269,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7816627025604248,
'end': 1666743424.2379317,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.state')},
'cmd_58': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'273253.44 IOPS 133.42 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'273253.44 IOPS 133.42 MiB/s\n'
'[2022-10-26 04:15:15.273977] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:15:15.274047] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:15:15.274058] bdevperf '
'[2022-10-26 04:15:15.274067] '
'--no-shconf [2022-10-26 '
'04:15:15.274075] -l 0 [2022-10-26 '
'04:15:15.274082] --huge-unlink '
'[2022-10-26 04:15:15.274091] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:15:15.274098] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:15:15.274105] '
'--log-level=user1:6 [2022-10-26 '
'04:15:15.274120] --iova-mode=pa '
'[2022-10-26 04:15:15.274128] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:15:15.274135] '
'--match-allocations [2022-10-26 '
'04:15:15.274142] '
'--file-prefix=spdk_pid137864 '
'[2022-10-26 04:15:15.274150] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:15:15.412303] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:15:15.453532] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:15:15.454217] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.output'),
'state': {'begin': 1666743425.2605772,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.19258713722229,
'end': 1666743436.4531643,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.state')},
'cmd_59': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.output'),
'state': {'begin': 1666743436.4555464,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7853162288665771,
'end': 1666743437.2408626,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.state')},
'cmd_60': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'273765.54 IOPS 133.67 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'273765.54 IOPS 133.67 MiB/s\n'
'[2022-10-26 04:15:28.271983] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:15:28.272054] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:15:28.272065] bdevperf '
'[2022-10-26 04:15:28.272073] '
'--no-shconf [2022-10-26 '
'04:15:28.272081] -l 0 [2022-10-26 '
'04:15:28.272089] --huge-unlink '
'[2022-10-26 04:15:28.272098] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:15:28.272106] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:15:28.272114] '
'--log-level=user1:6 [2022-10-26 '
'04:15:28.272122] --iova-mode=pa '
'[2022-10-26 04:15:28.272130] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:15:28.272137] '
'--match-allocations [2022-10-26 '
'04:15:28.272145] '
'--file-prefix=spdk_pid137889 '
'[2022-10-26 04:15:28.272153] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:15:28.410856] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:15:28.457568] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:15:28.458231] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.output'),
'state': {'begin': 1666743438.2684667,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 2 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.178375244140625,
'end': 1666743449.446842,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.state')},
'cmd_61': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.output'),
'state': {'begin': 1666743449.4493637,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7808904647827148,
'end': 1666743450.2302542,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.state')},
'cmd_62': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'392406.39 IOPS 191.60 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'392406.39 IOPS 191.60 MiB/s\n'
'[2022-10-26 04:15:41.256135] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:15:41.256203] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:15:41.256213] bdevperf '
'[2022-10-26 04:15:41.256222] '
'--no-shconf [2022-10-26 '
'04:15:41.256230] -l 0 [2022-10-26 '
'04:15:41.256238] --huge-unlink '
'[2022-10-26 04:15:41.256246] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:15:41.256253] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:15:41.256260] '
'--log-level=user1:6 [2022-10-26 '
'04:15:41.256268] --iova-mode=pa '
'[2022-10-26 04:15:41.256276] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:15:41.256283] '
'--match-allocations [2022-10-26 '
'04:15:41.256290] '
'--file-prefix=spdk_pid137914 '
'[2022-10-26 04:15:41.256298] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:15:41.394453] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:15:41.441506] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:15:41.442191] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.output'),
'state': {'begin': 1666743451.2566087,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.140043497085571,
'end': 1666743462.3966522,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.state')},
'cmd_63': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.output'),
'state': {'begin': 1666743462.4036922,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7801389694213867,
'end': 1666743463.1838312,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.state')},
'cmd_64': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'392406.89 IOPS 191.60 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'392406.89 IOPS 191.60 MiB/s\n'
'[2022-10-26 04:15:54.215274] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:15:54.215345] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:15:54.215356] bdevperf '
'[2022-10-26 04:15:54.215365] '
'--no-shconf [2022-10-26 '
'04:15:54.215374] -l 0 [2022-10-26 '
'04:15:54.215382] --huge-unlink '
'[2022-10-26 04:15:54.215390] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:15:54.215398] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:15:54.215406] '
'--log-level=user1:6 [2022-10-26 '
'04:15:54.215414] --iova-mode=pa '
'[2022-10-26 04:15:54.215423] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:15:54.215431] '
'--match-allocations [2022-10-26 '
'04:15:54.215439] '
'--file-prefix=spdk_pid137939 '
'[2022-10-26 04:15:54.215448] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:15:54.353784] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:15:54.405744] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:15:54.406405] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.output'),
'state': {'begin': 1666743464.210217,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.134780406951904,
'end': 1666743475.3449974,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.state')},
'cmd_65': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.output'),
'state': {'begin': 1666743475.3473835,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.787315845489502,
'end': 1666743476.1346993,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.state')},
'cmd_66': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'392071.55 IOPS 191.44 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'392071.55 IOPS 191.44 MiB/s\n'
'[2022-10-26 04:16:07.163390] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:16:07.163460] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:16:07.163471] bdevperf '
'[2022-10-26 04:16:07.163480] '
'--no-shconf [2022-10-26 '
'04:16:07.163488] -l 0 [2022-10-26 '
'04:16:07.163496] --huge-unlink '
'[2022-10-26 04:16:07.163503] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:16:07.163511] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:16:07.163518] '
'--log-level=user1:6 [2022-10-26 '
'04:16:07.163526] --iova-mode=pa '
'[2022-10-26 04:16:07.163533] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:16:07.163541] '
'--match-allocations [2022-10-26 '
'04:16:07.163548] '
'--file-prefix=spdk_pid137964 '
'[2022-10-26 04:16:07.163555] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:16:07.302310] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:16:07.345541] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:16:07.346210] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.output'),
'state': {'begin': 1666743477.1620119,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.115491151809692,
'end': 1666743488.277503,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.state')},
'cmd_67': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.output'),
'state': {'begin': 1666743488.2798777,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7826259136199951,
'end': 1666743489.0625036,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.state')},
'cmd_68': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'430379.28 IOPS 210.15 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'430379.28 IOPS 210.15 MiB/s\n'
'[2022-10-26 04:16:20.087240] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:16:20.087312] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:16:20.087323] bdevperf '
'[2022-10-26 04:16:20.087332] '
'--no-shconf [2022-10-26 '
'04:16:20.087340] -l 0 [2022-10-26 '
'04:16:20.087348] --huge-unlink '
'[2022-10-26 04:16:20.087357] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:16:20.087364] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:16:20.087372] '
'--log-level=user1:6 [2022-10-26 '
'04:16:20.087380] --iova-mode=pa '
'[2022-10-26 04:16:20.087388] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:16:20.087396] '
'--match-allocations [2022-10-26 '
'04:16:20.087404] '
'--file-prefix=spdk_pid137989 '
'[2022-10-26 04:16:20.087413] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:16:20.225818] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:16:20.269590] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:16:20.270271] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.output'),
'state': {'begin': 1666743490.08891,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.103523015975952,
'end': 1666743501.192433,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.state')},
'cmd_69': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.output'),
'state': {'begin': 1666743501.1953654,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.784801721572876,
'end': 1666743501.9801672,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.state')},
'cmd_70': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'431736.81 IOPS 210.81 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'431736.81 IOPS 210.81 MiB/s\n'
'[2022-10-26 04:16:33.002825] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:16:33.002895] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:16:33.002905] bdevperf '
'[2022-10-26 04:16:33.002915] '
'--no-shconf [2022-10-26 '
'04:16:33.002923] -l 0 [2022-10-26 '
'04:16:33.002930] --huge-unlink '
'[2022-10-26 04:16:33.002938] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:16:33.002946] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:16:33.002954] '
'--log-level=user1:6 [2022-10-26 '
'04:16:33.002962] --iova-mode=pa '
'[2022-10-26 04:16:33.002970] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:16:33.002977] '
'--match-allocations [2022-10-26 '
'04:16:33.002985] '
'--file-prefix=spdk_pid138014 '
'[2022-10-26 04:16:33.002992] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:16:33.141476] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:16:33.189533] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:16:33.190204] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.output'),
'state': {'begin': 1666743503.006869,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.098368167877197,
'end': 1666743514.1052372,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.state')},
'cmd_71': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.output'),
'state': {'begin': 1666743514.1128907,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7827651500701904,
'end': 1666743514.8956559,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.state')},
'cmd_72': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'428711.29 IOPS 209.33 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'428711.29 IOPS 209.33 MiB/s\n'
'[2022-10-26 04:16:45.923337] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:16:45.923406] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:16:45.923416] bdevperf '
'[2022-10-26 04:16:45.923425] '
'--no-shconf [2022-10-26 '
'04:16:45.923433] -l 0 [2022-10-26 '
'04:16:45.923441] --huge-unlink '
'[2022-10-26 04:16:45.923449] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:16:45.923456] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:16:45.923464] '
'--log-level=user1:6 [2022-10-26 '
'04:16:45.923471] --iova-mode=pa '
'[2022-10-26 04:16:45.923479] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:16:45.923486] '
'--match-allocations [2022-10-26 '
'04:16:45.923494] '
'--file-prefix=spdk_pid138039 '
'[2022-10-26 04:16:45.923501] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:16:46.062040] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:16:46.101512] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:16:46.102194] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.output'),
'state': {'begin': 1666743515.9263752,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.100262880325317,
'end': 1666743527.026638,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.state')},
'cmd_73': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.output'),
'state': {'begin': 1666743527.0294874,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7903826236724854,
'end': 1666743527.81987,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.state')},
'cmd_74': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'485865.81 IOPS 237.24 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'485865.81 IOPS 237.24 MiB/s\n'
'[2022-10-26 04:16:58.841718] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:16:58.841789] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:16:58.841800] bdevperf '
'[2022-10-26 04:16:58.841809] '
'--no-shconf [2022-10-26 '
'04:16:58.841817] -l 0 [2022-10-26 '
'04:16:58.841825] --huge-unlink '
'[2022-10-26 04:16:58.841833] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:16:58.841840] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:16:58.841848] '
'--log-level=user1:6 [2022-10-26 '
'04:16:58.841855] --iova-mode=pa '
'[2022-10-26 04:16:58.841863] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:16:58.841870] '
'--match-allocations [2022-10-26 '
'04:16:58.841878] '
'--file-prefix=spdk_pid138064 '
'[2022-10-26 04:16:58.841885] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:16:58.980698] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:16:59.025507] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:16:59.026173] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.output'),
'state': {'begin': 1666743528.8436036,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.105374336242676,
'end': 1666743539.948978,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.state')},
'cmd_75': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.output'),
'state': {'begin': 1666743539.9519782,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7835664749145508,
'end': 1666743540.7355447,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.state')},
'cmd_76': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'482781.41 IOPS 235.73 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'482781.41 IOPS 235.73 MiB/s\n'
'[2022-10-26 04:17:11.766933] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:17:11.767002] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:17:11.767013] bdevperf '
'[2022-10-26 04:17:11.767022] '
'--no-shconf [2022-10-26 '
'04:17:11.767030] -l 0 [2022-10-26 '
'04:17:11.767038] --huge-unlink '
'[2022-10-26 04:17:11.767047] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:17:11.767055] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:17:11.767062] '
'--log-level=user1:6 [2022-10-26 '
'04:17:11.767071] --iova-mode=pa '
'[2022-10-26 04:17:11.767079] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:17:11.767087] '
'--match-allocations [2022-10-26 '
'04:17:11.767095] '
'--file-prefix=spdk_pid138092 '
'[2022-10-26 04:17:11.767103] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:17:11.905315] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:17:11.945540] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:17:11.946218] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.output'),
'state': {'begin': 1666743541.7653217,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.098800420761108,
'end': 1666743552.8641222,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.state')},
'cmd_77': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.output'),
'state': {'begin': 1666743552.8684063,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7847492694854736,
'end': 1666743553.6531556,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.state')},
'cmd_78': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'482648.51 IOPS 235.67 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'482648.51 IOPS 235.67 MiB/s\n'
'[2022-10-26 04:17:24.672976] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:17:24.673047] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:17:24.673057] bdevperf '
'[2022-10-26 04:17:24.673066] '
'--no-shconf [2022-10-26 '
'04:17:24.673074] -l 0 [2022-10-26 '
'04:17:24.673082] --huge-unlink '
'[2022-10-26 04:17:24.673089] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:17:24.673097] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:17:24.673104] '
'--log-level=user1:6 [2022-10-26 '
'04:17:24.673112] --iova-mode=pa '
'[2022-10-26 04:17:24.673119] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:17:24.673126] '
'--match-allocations [2022-10-26 '
'04:17:24.673134] '
'--file-prefix=spdk_pid138117 '
'[2022-10-26 04:17:24.673141] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:17:24.812469] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:17:24.861970] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:17:24.862650] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.output'),
'state': {'begin': 1666743554.6805289,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.108688592910767,
'end': 1666743565.7892175,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.state')},
'cmd_79': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.output'),
'state': {'begin': 1666743565.7918122,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7834320068359375,
'end': 1666743566.5752442,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.state')},
'cmd_80': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'479021.22 IOPS 233.90 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'479021.22 IOPS 233.90 MiB/s\n'
'[2022-10-26 04:17:37.615480] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:17:37.615550] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:17:37.615561] bdevperf '
'[2022-10-26 04:17:37.615570] '
'--no-shconf [2022-10-26 '
'04:17:37.615578] -l 0 [2022-10-26 '
'04:17:37.615586] --huge-unlink '
'[2022-10-26 04:17:37.615594] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:17:37.615601] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:17:37.615608] '
'--log-level=user1:6 [2022-10-26 '
'04:17:37.615616] --iova-mode=pa '
'[2022-10-26 04:17:37.615623] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:17:37.615631] '
'--match-allocations [2022-10-26 '
'04:17:37.615639] '
'--file-prefix=spdk_pid138142 '
'[2022-10-26 04:17:37.615647] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:17:37.754205] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:17:37.797538] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:17:37.798220] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.output'),
'state': {'begin': 1666743567.6054509,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.18461012840271,
'end': 1666743578.790061,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.state')},
'cmd_81': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.output'),
'state': {'begin': 1666743578.7931554,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7960965633392334,
'end': 1666743579.589252,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.state')},
'cmd_82': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'479430.62 IOPS 234.10 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'479430.62 IOPS 234.10 MiB/s\n'
'[2022-10-26 04:17:50.615507] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:17:50.615575] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:17:50.615586] bdevperf '
'[2022-10-26 04:17:50.615594] '
'--no-shconf [2022-10-26 '
'04:17:50.615602] -l 0 [2022-10-26 '
'04:17:50.615610] --huge-unlink '
'[2022-10-26 04:17:50.615617] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:17:50.615625] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:17:50.615632] '
'--log-level=user1:6 [2022-10-26 '
'04:17:50.615640] --iova-mode=pa '
'[2022-10-26 04:17:50.615648] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:17:50.615655] '
'--match-allocations [2022-10-26 '
'04:17:50.615663] '
'--file-prefix=spdk_pid138167 '
'[2022-10-26 04:17:50.615670] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:17:50.752773] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:17:50.805554] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:17:50.806237] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.output'),
'state': {'begin': 1666743580.6223962,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.173489809036255,
'end': 1666743591.795886,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.state')},
'cmd_83': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.output'),
'state': {'begin': 1666743591.7985127,
'cmd': 'rm '
'/tmp/spdk_bdev_aio_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7782433032989502,
'end': 1666743592.576756,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.state')},
'cmd_84': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'480153.62 IOPS 234.45 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'480153.62 IOPS 234.45 MiB/s\n'
'[2022-10-26 04:18:03.607723] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:18:03.607800] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:18:03.607811] bdevperf '
'[2022-10-26 04:18:03.607820] '
'--no-shconf [2022-10-26 '
'04:18:03.607828] -l 0 [2022-10-26 '
'04:18:03.607836] --huge-unlink '
'[2022-10-26 04:18:03.607844] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:18:03.607852] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:18:03.607859] '
'--log-level=user1:6 [2022-10-26 '
'04:18:03.607868] --iova-mode=pa '
'[2022-10-26 04:18:03.607876] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:18:03.607884] '
'--match-allocations [2022-10-26 '
'04:18:03.607892] '
'--file-prefix=spdk_pid138192 '
'[2022-10-26 04:18:03.607900] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:18:03.746385] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:18:03.789546] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:18:03.790227] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.output'),
'state': {'begin': 1666743593.607161,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_aio_foo.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.168599128723145,
'end': 1666743604.7757602,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.state')},
'cmd_85': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.output'),
'state': {'begin': 1666743604.7787178,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7873766422271729,
'end': 1666743605.5660944,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.state')},
'cmd_86': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'430950.51 IOPS 210.43 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'430950.51 IOPS 210.43 MiB/s\n'
'[2022-10-26 04:18:16.597404] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:18:16.597472] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:18:16.597483] bdevperf '
'[2022-10-26 04:18:16.597491] '
'--no-shconf [2022-10-26 '
'04:18:16.597499] -l 0 [2022-10-26 '
'04:18:16.597507] --huge-unlink '
'[2022-10-26 04:18:16.597516] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:18:16.597524] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:18:16.597531] '
'--log-level=user1:6 [2022-10-26 '
'04:18:16.597557] --iova-mode=pa '
'[2022-10-26 04:18:16.597567] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:18:16.597575] '
'--match-allocations [2022-10-26 '
'04:18:16.597583] '
'--file-prefix=spdk_pid138217 '
'[2022-10-26 04:18:16.597599] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:18:16.735774] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:18:16.777548] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:18:16.778223] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.output'),
'state': {'begin': 1666743606.5988963,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.173280954360962,
'end': 1666743617.7721772,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.state')},
'cmd_87': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.output'),
'state': {'begin': 1666743617.7751496,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7903523445129395,
'end': 1666743618.565502,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.state')},
'cmd_88': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'429672.51 IOPS 209.80 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'429672.51 IOPS 209.80 MiB/s\n'
'[2022-10-26 04:18:29.614303] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:18:29.614376] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:18:29.614387] bdevperf '
'[2022-10-26 04:18:29.614395] '
'--no-shconf [2022-10-26 '
'04:18:29.614403] -l 0 [2022-10-26 '
'04:18:29.614411] --huge-unlink '
'[2022-10-26 04:18:29.614419] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:18:29.614426] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:18:29.614433] '
'--log-level=user1:6 [2022-10-26 '
'04:18:29.614441] --iova-mode=pa '
'[2022-10-26 04:18:29.614448] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:18:29.614455] '
'--match-allocations [2022-10-26 '
'04:18:29.614463] '
'--file-prefix=spdk_pid138243 '
'[2022-10-26 04:18:29.614470] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:18:29.753633] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:18:29.801578] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:18:29.802241] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.output'),
'state': {'begin': 1666743619.6014123,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.215760231018066,
'end': 1666743630.8171725,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.state')},
'cmd_89': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.output'),
'state': {'begin': 1666743630.8198419,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7805182933807373,
'end': 1666743631.6003602,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.state')},
'cmd_90': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'430511.26 IOPS 210.21 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'430511.26 IOPS 210.21 MiB/s\n'
'[2022-10-26 04:18:42.616257] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:18:42.616325] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:18:42.616336] bdevperf '
'[2022-10-26 04:18:42.616345] '
'--no-shconf [2022-10-26 '
'04:18:42.616353] -l 0 [2022-10-26 '
'04:18:42.616362] --huge-unlink '
'[2022-10-26 04:18:42.616370] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:18:42.616378] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:18:42.616386] '
'--log-level=user1:6 [2022-10-26 '
'04:18:42.616394] --iova-mode=pa '
'[2022-10-26 04:18:42.616410] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:18:42.616419] '
'--match-allocations [2022-10-26 '
'04:18:42.616427] '
'--file-prefix=spdk_pid138269 '
'[2022-10-26 04:18:42.616435] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:18:42.754809] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:18:42.801477] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:18:42.802142] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.output'),
'state': {'begin': 1666743632.626331,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_libaio.conf '
'-q 4 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.178959846496582,
'end': 1666743643.805291,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.state')},
'cmd_91': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.output'),
'state': {'begin': 1666743643.8072298,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7829594612121582,
'end': 1666743644.5901892,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.state')},
'cmd_92': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'616823.81 IOPS 301.18 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'616823.81 IOPS 301.18 MiB/s\n'
'[2022-10-26 04:18:55.614634] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:18:55.614701] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:18:55.614712] bdevperf '
'[2022-10-26 04:18:55.614722] '
'--no-shconf [2022-10-26 '
'04:18:55.614730] -l 0 [2022-10-26 '
'04:18:55.614737] --huge-unlink '
'[2022-10-26 04:18:55.614745] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:18:55.614752] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:18:55.614759] '
'--log-level=user1:6 [2022-10-26 '
'04:18:55.614767] --iova-mode=pa '
'[2022-10-26 04:18:55.614774] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:18:55.614782] '
'--match-allocations [2022-10-26 '
'04:18:55.614789] '
'--file-prefix=spdk_pid138294 '
'[2022-10-26 04:18:55.614796] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:18:55.753382] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:18:55.793511] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:18:55.794175] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.output'),
'state': {'begin': 1666743645.6223195,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 8 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.114681482315063,
'end': 1666743656.737001,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.state')},
'cmd_93': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.output'),
'state': {'begin': 1666743656.7398794,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7825148105621338,
'end': 1666743657.5223942,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.state')},
'cmd_94': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'620874.40 IOPS 303.16 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'620874.40 IOPS 303.16 MiB/s\n'
'[2022-10-26 04:19:08.547339] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:19:08.547409] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:19:08.547421] bdevperf '
'[2022-10-26 04:19:08.547430] '
'--no-shconf [2022-10-26 '
'04:19:08.547438] -l 0 [2022-10-26 '
'04:19:08.547446] --huge-unlink '
'[2022-10-26 04:19:08.547454] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:19:08.547461] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:19:08.547469] '
'--log-level=user1:6 [2022-10-26 '
'04:19:08.547476] --iova-mode=pa '
'[2022-10-26 04:19:08.547484] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:19:08.547492] '
'--match-allocations [2022-10-26 '
'04:19:08.547499] '
'--file-prefix=spdk_pid138319 '
'[2022-10-26 04:19:08.547506] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:19:08.685968] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:19:08.729521] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:19:08.730182] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.output'),
'state': {'begin': 1666743658.5476117,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 8 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.11674427986145,
'end': 1666743669.664356,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.state')},
'cmd_95': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.output'),
'state': {'begin': 1666743669.666656,
'cmd': 'rm '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7816565036773682,
'end': 1666743670.4483125,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.state')},
'cmd_96': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'618850.10 IOPS 302.17 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'618850.10 IOPS 302.17 MiB/s\n'
'[2022-10-26 04:19:21.485327] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:19:21.485385] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:19:21.485394] bdevperf '
'[2022-10-26 04:19:21.485400] '
'--no-shconf [2022-10-26 '
'04:19:21.485406] -l 0 [2022-10-26 '
'04:19:21.485411] --huge-unlink '
'[2022-10-26 04:19:21.485418] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:19:21.485423] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:19:21.485429] '
'--log-level=user1:6 [2022-10-26 '
'04:19:21.485434] --iova-mode=pa '
'[2022-10-26 04:19:21.485440] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:19:21.485445] '
'--match-allocations [2022-10-26 '
'04:19:21.485451] '
'--file-prefix=spdk_pid138345 '
'[2022-10-26 04:19:21.485456] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:19:21.617444] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:19:21.673483] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:19:21.674157] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.output'),
'state': {'begin': 1666743671.4852216,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
'-q 8 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.12417483329773,
'end': 1666743682.6093965,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.state')},
'cmd_97': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.output'),
'state': {'begin': 1666743682.611976,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7869632244110107,
'end': 1666743683.3989391,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.state')},
'cmd_98': {'output': 'Running I/O for 10 seconds...\n'
'\n'
' Job: Nvme0n1 (Core Mask 0x1)\n'
'\t Nvme0n1 : '
'518690.13 IOPS 253.27 MiB/s\n'
'\n'
' '
'=============================================================\n'
'\n'
' Total : '
'518690.13 IOPS 253.27 MiB/s\n'
'[2022-10-26 04:19:34.430663] '
'Starting SPDK v23.01-pre git sha1 '
'fd41a2937 / DPDK 22.07.0 '
'initialization...\n'
'[2022-10-26 04:19:34.430733] [ '
'DPDK EAL parameters: [2022-10-26 '
'04:19:34.430744] bdevperf '
'[2022-10-26 04:19:34.430753] '
'--no-shconf [2022-10-26 '
'04:19:34.430761] -l 0 [2022-10-26 '
'04:19:34.430769] --huge-unlink '
'[2022-10-26 04:19:34.430776] '
'--log-level=lib.eal:6 [2022-10-26 '
'04:19:34.430783] '
'--log-level=lib.cryptodev:5 '
'[2022-10-26 04:19:34.430790] '
'--log-level=user1:6 [2022-10-26 '
'04:19:34.430798] --iova-mode=pa '
'[2022-10-26 04:19:34.430805] '
'--base-virtaddr=0x200000000000 '
'[2022-10-26 04:19:34.430812] '
'--match-allocations [2022-10-26 '
'04:19:34.430819] '
'--file-prefix=spdk_pid138371 '
'[2022-10-26 04:19:34.430826] ]\n'
'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n'
'[2022-10-26 04:19:34.570480] '
'app.c: 586:spdk_app_start: '
'*NOTICE*: Total cores available: '
'1\n'
'[2022-10-26 04:19:34.614037] '
'reactor.c: 926:reactor_run: '
'*NOTICE*: Reactor started on core '
'0\n'
'[2022-10-26 04:19:34.614728] '
'accel_sw.c: '
'461:sw_accel_module_init: '
'*NOTICE*: Accel framework '
'software module initialized.\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.output'),
'state': {'begin': 1666743684.4286978,
'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
'--json '
'/tmp/spdk_bdev_uring_foo.conf '
'-q 8 -o 512 -w randread -t '
"10 -m '[0]'",
'cwd': 'None',
'elapsed': 11.109190702438354,
'end': 1666743695.5378885,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.state')},
'cmd_99': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.output'),
'state': {'begin': 1666743695.5408466,
'cmd': 'rm '
'/tmp/spdk_bdev_uring_foo.conf '
'|| true',
'cwd': 'None',
'elapsed': 0.7842028141021729,
'end': 1666743696.3250494,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.state')}}},
'id': '001_bdevperf',
'name': 'bdevperf',
'nr': 1,
'status': {'elapsed': 777.7638144493103,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'bdevperf',
'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['512'], 'repetitions': 3}}
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 171834.77 IOPS 83.90 MiB/s
=============================================================
Total : 171834.77 IOPS 83.90 MiB/s
[2022-10-26 04:09:12.283172] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:09:12.283242] [ DPDK EAL parameters: [2022-10-26 04:09:12.283253] bdevperf [2022-10-26 04:09:12.283263] --no-shconf [2022-10-26 04:09:12.283271] -l 0 [2022-10-26 04:09:12.283280] --huge-unlink [2022-10-26 04:09:12.283289] --log-level=lib.eal:6 [2022-10-26 04:09:12.283296] --log-level=lib.cryptodev:5 [2022-10-26 04:09:12.283304] --log-level=user1:6 [2022-10-26 04:09:12.283312] --iova-mode=pa [2022-10-26 04:09:12.283320] --base-virtaddr=0x200000000000 [2022-10-26 04:09:12.283328] --match-allocations [2022-10-26 04:09:12.283335] --file-prefix=spdk_pid137152 [2022-10-26 04:09:12.283344] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:09:12.421601] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:09:12.473509] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:09:12.474184] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 172633.05 IOPS 84.29 MiB/s
=============================================================
Total : 172633.05 IOPS 84.29 MiB/s
[2022-10-26 04:09:25.202810] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:09:25.202878] [ DPDK EAL parameters: [2022-10-26 04:09:25.202890] bdevperf [2022-10-26 04:09:25.202898] --no-shconf [2022-10-26 04:09:25.202906] -l 0 [2022-10-26 04:09:25.202914] --huge-unlink [2022-10-26 04:09:25.202921] --log-level=lib.eal:6 [2022-10-26 04:09:25.202930] --log-level=lib.cryptodev:5 [2022-10-26 04:09:25.202937] --log-level=user1:6 [2022-10-26 04:09:25.202944] --iova-mode=pa [2022-10-26 04:09:25.202952] --base-virtaddr=0x200000000000 [2022-10-26 04:09:25.202959] --match-allocations [2022-10-26 04:09:25.202967] --file-prefix=spdk_pid137177 [2022-10-26 04:09:25.202974] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:09:25.341687] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:09:25.389529] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:09:25.390195] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 171552.75 IOPS 83.77 MiB/s
=============================================================
Total : 171552.75 IOPS 83.77 MiB/s
[2022-10-26 04:09:38.130730] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:09:38.130800] [ DPDK EAL parameters: [2022-10-26 04:09:38.130811] bdevperf [2022-10-26 04:09:38.130819] --no-shconf [2022-10-26 04:09:38.130828] -l 0 [2022-10-26 04:09:38.130836] --huge-unlink [2022-10-26 04:09:38.130845] --log-level=lib.eal:6 [2022-10-26 04:09:38.130853] --log-level=lib.cryptodev:5 [2022-10-26 04:09:38.130860] --log-level=user1:6 [2022-10-26 04:09:38.130868] --iova-mode=pa [2022-10-26 04:09:38.130876] --base-virtaddr=0x200000000000 [2022-10-26 04:09:38.130885] --match-allocations [2022-10-26 04:09:38.130893] --file-prefix=spdk_pid137202 [2022-10-26 04:09:38.130901] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:09:38.269317] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:09:38.321522] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:09:38.322192] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 129651.27 IOPS 63.31 MiB/s
=============================================================
Total : 129651.27 IOPS 63.31 MiB/s
[2022-10-26 04:09:51.072421] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:09:51.072495] [ DPDK EAL parameters: [2022-10-26 04:09:51.072506] bdevperf [2022-10-26 04:09:51.072515] --no-shconf [2022-10-26 04:09:51.072524] -l 0 [2022-10-26 04:09:51.072532] --huge-unlink [2022-10-26 04:09:51.072541] --log-level=lib.eal:6 [2022-10-26 04:09:51.072549] --log-level=lib.cryptodev:5 [2022-10-26 04:09:51.072557] --log-level=user1:6 [2022-10-26 04:09:51.072566] --iova-mode=pa [2022-10-26 04:09:51.072574] --base-virtaddr=0x200000000000 [2022-10-26 04:09:51.072583] --match-allocations [2022-10-26 04:09:51.072591] --file-prefix=spdk_pid137229 [2022-10-26 04:09:51.072600] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:09:51.210930] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:09:51.257547] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:09:51.258212] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 129366.45 IOPS 63.17 MiB/s
=============================================================
Total : 129366.45 IOPS 63.17 MiB/s
[2022-10-26 04:10:03.980551] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:03.980620] [ DPDK EAL parameters: [2022-10-26 04:10:03.980631] bdevperf [2022-10-26 04:10:03.980640] --no-shconf [2022-10-26 04:10:03.980648] -l 0 [2022-10-26 04:10:03.980655] --huge-unlink [2022-10-26 04:10:03.980663] --log-level=lib.eal:6 [2022-10-26 04:10:03.980671] --log-level=lib.cryptodev:5 [2022-10-26 04:10:03.980678] --log-level=user1:6 [2022-10-26 04:10:03.980686] --iova-mode=pa [2022-10-26 04:10:03.980693] --base-virtaddr=0x200000000000 [2022-10-26 04:10:03.980700] --match-allocations [2022-10-26 04:10:03.980708] --file-prefix=spdk_pid137254 [2022-10-26 04:10:03.980716] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:04.119553] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:04.161522] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:04.162184] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 519280.46 IOPS 253.55 MiB/s
=============================================================
Total : 519280.46 IOPS 253.55 MiB/s
[2022-10-26 04:19:47.360269] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:19:47.360339] [ DPDK EAL parameters: [2022-10-26 04:19:47.360350] bdevperf [2022-10-26 04:19:47.360359] --no-shconf [2022-10-26 04:19:47.360367] -l 0 [2022-10-26 04:19:47.360375] --huge-unlink [2022-10-26 04:19:47.360384] --log-level=lib.eal:6 [2022-10-26 04:19:47.360392] --log-level=lib.cryptodev:5 [2022-10-26 04:19:47.360399] --log-level=user1:6 [2022-10-26 04:19:47.360407] --iova-mode=pa [2022-10-26 04:19:47.360416] --base-virtaddr=0x200000000000 [2022-10-26 04:19:47.360423] --match-allocations [2022-10-26 04:19:47.360431] --file-prefix=spdk_pid138396 [2022-10-26 04:19:47.360440] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:19:47.498685] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:19:47.549512] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:19:47.550170] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 520408.18 IOPS 254.11 MiB/s
=============================================================
Total : 520408.18 IOPS 254.11 MiB/s
[2022-10-26 04:20:00.275836] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:00.275907] [ DPDK EAL parameters: [2022-10-26 04:20:00.275918] bdevperf [2022-10-26 04:20:00.275926] --no-shconf [2022-10-26 04:20:00.275934] -l 0 [2022-10-26 04:20:00.275942] --huge-unlink [2022-10-26 04:20:00.275949] --log-level=lib.eal:6 [2022-10-26 04:20:00.275957] --log-level=lib.cryptodev:5 [2022-10-26 04:20:00.275964] --log-level=user1:6 [2022-10-26 04:20:00.275972] --iova-mode=pa [2022-10-26 04:20:00.275979] --base-virtaddr=0x200000000000 [2022-10-26 04:20:00.275987] --match-allocations [2022-10-26 04:20:00.275994] --file-prefix=spdk_pid138421 [2022-10-26 04:20:00.276001] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:00.414472] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:00.453507] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:00.454165] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 534233.37 IOPS 260.86 MiB/s
=============================================================
Total : 534233.37 IOPS 260.86 MiB/s
[2022-10-26 04:20:13.190404] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:13.190474] [ DPDK EAL parameters: [2022-10-26 04:20:13.190484] bdevperf [2022-10-26 04:20:13.190494] --no-shconf [2022-10-26 04:20:13.190502] -l 0 [2022-10-26 04:20:13.190509] --huge-unlink [2022-10-26 04:20:13.190517] --log-level=lib.eal:6 [2022-10-26 04:20:13.190525] --log-level=lib.cryptodev:5 [2022-10-26 04:20:13.190532] --log-level=user1:6 [2022-10-26 04:20:13.190539] --iova-mode=pa [2022-10-26 04:20:13.190546] --base-virtaddr=0x200000000000 [2022-10-26 04:20:13.190554] --match-allocations [2022-10-26 04:20:13.190561] --file-prefix=spdk_pid138447 [2022-10-26 04:20:13.190568] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:13.328990] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:13.369543] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:13.370218] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 527043.53 IOPS 257.35 MiB/s
=============================================================
Total : 527043.53 IOPS 257.35 MiB/s
[2022-10-26 04:20:26.114969] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:26.115039] [ DPDK EAL parameters: [2022-10-26 04:20:26.115050] bdevperf [2022-10-26 04:20:26.115059] --no-shconf [2022-10-26 04:20:26.115067] -l 0 [2022-10-26 04:20:26.115075] --huge-unlink [2022-10-26 04:20:26.115083] --log-level=lib.eal:6 [2022-10-26 04:20:26.115090] --log-level=lib.cryptodev:5 [2022-10-26 04:20:26.115097] --log-level=user1:6 [2022-10-26 04:20:26.115105] --iova-mode=pa [2022-10-26 04:20:26.115123] --base-virtaddr=0x200000000000 [2022-10-26 04:20:26.115131] --match-allocations [2022-10-26 04:20:26.115138] --file-prefix=spdk_pid138472 [2022-10-26 04:20:26.115146] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:26.253677] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:26.301556] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:26.302227] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 530831.57 IOPS 259.20 MiB/s
=============================================================
Total : 530831.57 IOPS 259.20 MiB/s
[2022-10-26 04:20:39.084853] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:39.084921] [ DPDK EAL parameters: [2022-10-26 04:20:39.084931] bdevperf [2022-10-26 04:20:39.084940] --no-shconf [2022-10-26 04:20:39.084949] -l 0 [2022-10-26 04:20:39.084957] --huge-unlink [2022-10-26 04:20:39.084966] --log-level=lib.eal:6 [2022-10-26 04:20:39.084974] --log-level=lib.cryptodev:5 [2022-10-26 04:20:39.084981] --log-level=user1:6 [2022-10-26 04:20:39.084990] --iova-mode=pa [2022-10-26 04:20:39.084998] --base-virtaddr=0x200000000000 [2022-10-26 04:20:39.085006] --match-allocations [2022-10-26 04:20:39.085015] --file-prefix=spdk_pid138497 [2022-10-26 04:20:39.085023] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:39.223403] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:39.269614] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:39.270283] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 533336.85 IOPS 260.42 MiB/s
=============================================================
Total : 533336.85 IOPS 260.42 MiB/s
[2022-10-26 04:20:52.015515] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:52.015584] [ DPDK EAL parameters: [2022-10-26 04:20:52.015595] bdevperf [2022-10-26 04:20:52.015604] --no-shconf [2022-10-26 04:20:52.015612] -l 0 [2022-10-26 04:20:52.015620] --huge-unlink [2022-10-26 04:20:52.015628] --log-level=lib.eal:6 [2022-10-26 04:20:52.015636] --log-level=lib.cryptodev:5 [2022-10-26 04:20:52.015643] --log-level=user1:6 [2022-10-26 04:20:52.015650] --iova-mode=pa [2022-10-26 04:20:52.015658] --base-virtaddr=0x200000000000 [2022-10-26 04:20:52.015665] --match-allocations [2022-10-26 04:20:52.015673] --file-prefix=spdk_pid138522 [2022-10-26 04:20:52.015680] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:52.154077] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:52.193585] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:52.194258] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 536250.58 IOPS 261.84 MiB/s
=============================================================
Total : 536250.58 IOPS 261.84 MiB/s
[2022-10-26 04:21:05.002701] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:05.002771] [ DPDK EAL parameters: [2022-10-26 04:21:05.002782] bdevperf [2022-10-26 04:21:05.002791] --no-shconf [2022-10-26 04:21:05.002800] -l 0 [2022-10-26 04:21:05.002808] --huge-unlink [2022-10-26 04:21:05.002816] --log-level=lib.eal:6 [2022-10-26 04:21:05.002823] --log-level=lib.cryptodev:5 [2022-10-26 04:21:05.002831] --log-level=user1:6 [2022-10-26 04:21:05.002846] --iova-mode=pa [2022-10-26 04:21:05.002854] --base-virtaddr=0x200000000000 [2022-10-26 04:21:05.002861] --match-allocations [2022-10-26 04:21:05.002870] --file-prefix=spdk_pid138547 [2022-10-26 04:21:05.002878] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:05.141323] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:05.185509] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:05.186190] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 534521.72 IOPS 261.00 MiB/s
=============================================================
Total : 534521.72 IOPS 261.00 MiB/s
[2022-10-26 04:21:17.969960] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:17.970030] [ DPDK EAL parameters: [2022-10-26 04:21:17.970041] bdevperf [2022-10-26 04:21:17.970050] --no-shconf [2022-10-26 04:21:17.970058] -l 0 [2022-10-26 04:21:17.970066] --huge-unlink [2022-10-26 04:21:17.970074] --log-level=lib.eal:6 [2022-10-26 04:21:17.970081] --log-level=lib.cryptodev:5 [2022-10-26 04:21:17.970088] --log-level=user1:6 [2022-10-26 04:21:17.970096] --iova-mode=pa [2022-10-26 04:21:17.970103] --base-virtaddr=0x200000000000 [2022-10-26 04:21:17.970111] --match-allocations [2022-10-26 04:21:17.970118] --file-prefix=spdk_pid138572 [2022-10-26 04:21:17.970126] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:18.108770] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:18.149565] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:18.150241] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 510627.58 IOPS 249.33 MiB/s
=============================================================
Total : 510627.58 IOPS 249.33 MiB/s
[2022-10-26 04:21:30.943091] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:30.943175] [ DPDK EAL parameters: [2022-10-26 04:21:30.943186] bdevperf [2022-10-26 04:21:30.943195] --no-shconf [2022-10-26 04:21:30.943204] -l 0 [2022-10-26 04:21:30.943211] --huge-unlink [2022-10-26 04:21:30.943220] --log-level=lib.eal:6 [2022-10-26 04:21:30.943227] --log-level=lib.cryptodev:5 [2022-10-26 04:21:30.943235] --log-level=user1:6 [2022-10-26 04:21:30.943243] --iova-mode=pa [2022-10-26 04:21:30.943251] --base-virtaddr=0x200000000000 [2022-10-26 04:21:30.943259] --match-allocations [2022-10-26 04:21:30.943267] --file-prefix=spdk_pid138597 [2022-10-26 04:21:30.943275] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:31.081602] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:31.121496] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:31.122161] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 507405.68 IOPS 247.76 MiB/s
=============================================================
Total : 507405.68 IOPS 247.76 MiB/s
[2022-10-26 04:21:43.951587] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:43.951660] [ DPDK EAL parameters: [2022-10-26 04:21:43.951671] bdevperf [2022-10-26 04:21:43.951680] --no-shconf [2022-10-26 04:21:43.951688] -l 0 [2022-10-26 04:21:43.951696] --huge-unlink [2022-10-26 04:21:43.951704] --log-level=lib.eal:6 [2022-10-26 04:21:43.951712] --log-level=lib.cryptodev:5 [2022-10-26 04:21:43.951719] --log-level=user1:6 [2022-10-26 04:21:43.951726] --iova-mode=pa [2022-10-26 04:21:43.951734] --base-virtaddr=0x200000000000 [2022-10-26 04:21:43.951741] --match-allocations [2022-10-26 04:21:43.951749] --file-prefix=spdk_pid138623 [2022-10-26 04:21:43.951756] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:44.090207] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:44.129577] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:44.130245] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 129066.85 IOPS 63.02 MiB/s
=============================================================
Total : 129066.85 IOPS 63.02 MiB/s
[2022-10-26 04:10:16.900959] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:16.901031] [ DPDK EAL parameters: [2022-10-26 04:10:16.901042] bdevperf [2022-10-26 04:10:16.901050] --no-shconf [2022-10-26 04:10:16.901058] -l 0 [2022-10-26 04:10:16.901066] --huge-unlink [2022-10-26 04:10:16.901074] --log-level=lib.eal:6 [2022-10-26 04:10:16.901081] --log-level=lib.cryptodev:5 [2022-10-26 04:10:16.901088] --log-level=user1:6 [2022-10-26 04:10:16.901096] --iova-mode=pa [2022-10-26 04:10:16.901104] --base-virtaddr=0x200000000000 [2022-10-26 04:10:16.901111] --match-allocations [2022-10-26 04:10:16.901118] --file-prefix=spdk_pid137279 [2022-10-26 04:10:16.901126] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:17.039698] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:17.081578] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:17.082253] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 510139.78 IOPS 249.09 MiB/s
=============================================================
Total : 510139.78 IOPS 249.09 MiB/s
[2022-10-26 04:21:56.963424] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:56.963490] [ DPDK EAL parameters: [2022-10-26 04:21:56.963502] bdevperf [2022-10-26 04:21:56.963511] --no-shconf [2022-10-26 04:21:56.963520] -l 0 [2022-10-26 04:21:56.963527] --huge-unlink [2022-10-26 04:21:56.963536] --log-level=lib.eal:6 [2022-10-26 04:21:56.963543] --log-level=lib.cryptodev:5 [2022-10-26 04:21:56.963551] --log-level=user1:6 [2022-10-26 04:21:56.963559] --iova-mode=pa [2022-10-26 04:21:56.963567] --base-virtaddr=0x200000000000 [2022-10-26 04:21:56.963575] --match-allocations [2022-10-26 04:21:56.963583] --file-prefix=spdk_pid138648 [2022-10-26 04:21:56.963590] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:57.102363] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:57.145569] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:57.146250] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 147294.48 IOPS 71.92 MiB/s
=============================================================
Total : 147294.48 IOPS 71.92 MiB/s
[2022-10-26 04:10:29.810550] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:29.810618] [ DPDK EAL parameters: [2022-10-26 04:10:29.810628] bdevperf [2022-10-26 04:10:29.810637] --no-shconf [2022-10-26 04:10:29.810646] -l 0 [2022-10-26 04:10:29.810653] --huge-unlink [2022-10-26 04:10:29.810662] --log-level=lib.eal:6 [2022-10-26 04:10:29.810670] --log-level=lib.cryptodev:5 [2022-10-26 04:10:29.810677] --log-level=user1:6 [2022-10-26 04:10:29.810685] --iova-mode=pa [2022-10-26 04:10:29.810694] --base-virtaddr=0x200000000000 [2022-10-26 04:10:29.810701] --match-allocations [2022-10-26 04:10:29.810710] --file-prefix=spdk_pid137304 [2022-10-26 04:10:29.810718] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:29.949133] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:29.993590] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:29.994247] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 147811.38 IOPS 72.17 MiB/s
=============================================================
Total : 147811.38 IOPS 72.17 MiB/s
[2022-10-26 04:10:42.746033] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:42.746103] [ DPDK EAL parameters: [2022-10-26 04:10:42.746113] bdevperf [2022-10-26 04:10:42.746122] --no-shconf [2022-10-26 04:10:42.746130] -l 0 [2022-10-26 04:10:42.746139] --huge-unlink [2022-10-26 04:10:42.746146] --log-level=lib.eal:6 [2022-10-26 04:10:42.746154] --log-level=lib.cryptodev:5 [2022-10-26 04:10:42.746161] --log-level=user1:6 [2022-10-26 04:10:42.746169] --iova-mode=pa [2022-10-26 04:10:42.746176] --base-virtaddr=0x200000000000 [2022-10-26 04:10:42.746183] --match-allocations [2022-10-26 04:10:42.746191] --file-prefix=spdk_pid137329 [2022-10-26 04:10:42.746198] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:42.884755] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:42.934563] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:42.935300] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 147836.09 IOPS 72.19 MiB/s
=============================================================
Total : 147836.09 IOPS 72.19 MiB/s
[2022-10-26 04:10:55.660577] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:55.660666] [ DPDK EAL parameters: [2022-10-26 04:10:55.660678] bdevperf [2022-10-26 04:10:55.660688] --no-shconf [2022-10-26 04:10:55.660697] -l 0 [2022-10-26 04:10:55.660706] --huge-unlink [2022-10-26 04:10:55.660716] --log-level=lib.eal:6 [2022-10-26 04:10:55.660724] --log-level=lib.cryptodev:5 [2022-10-26 04:10:55.660733] --log-level=user1:6 [2022-10-26 04:10:55.660742] --iova-mode=pa [2022-10-26 04:10:55.660750] --base-virtaddr=0x200000000000 [2022-10-26 04:10:55.660759] --match-allocations [2022-10-26 04:10:55.660768] --file-prefix=spdk_pid137354 [2022-10-26 04:10:55.660776] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:55.802085] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:55.849530] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:55.850223] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 145829.56 IOPS 71.21 MiB/s
=============================================================
Total : 145829.56 IOPS 71.21 MiB/s
[2022-10-26 04:11:08.598776] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:11:08.598845] [ DPDK EAL parameters: [2022-10-26 04:11:08.598856] bdevperf [2022-10-26 04:11:08.598865] --no-shconf [2022-10-26 04:11:08.598874] -l 0 [2022-10-26 04:11:08.598882] --huge-unlink [2022-10-26 04:11:08.598891] --log-level=lib.eal:6 [2022-10-26 04:11:08.598898] --log-level=lib.cryptodev:5 [2022-10-26 04:11:08.598905] --log-level=user1:6 [2022-10-26 04:11:08.598915] --iova-mode=pa [2022-10-26 04:11:08.598923] --base-virtaddr=0x200000000000 [2022-10-26 04:11:08.598931] --match-allocations [2022-10-26 04:11:08.598939] --file-prefix=spdk_pid137380 [2022-10-26 04:11:08.598947] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:11:08.737667] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:11:08.785519] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:11:08.786203] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 157191.61 IOPS 76.75 MiB/s
=============================================================
Total : 157191.61 IOPS 76.75 MiB/s
[2022-10-26 04:11:21.593539] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:11:21.593610] [ DPDK EAL parameters: [2022-10-26 04:11:21.593621] bdevperf [2022-10-26 04:11:21.593629] --no-shconf [2022-10-26 04:11:21.593638] -l 0 [2022-10-26 04:11:21.593646] --huge-unlink [2022-10-26 04:11:21.593653] --log-level=lib.eal:6 [2022-10-26 04:11:21.593661] --log-level=lib.cryptodev:5 [2022-10-26 04:11:21.593669] --log-level=user1:6 [2022-10-26 04:11:21.593677] --iova-mode=pa [2022-10-26 04:11:21.593685] --base-virtaddr=0x200000000000 [2022-10-26 04:11:21.593693] --match-allocations [2022-10-26 04:11:21.593701] --file-prefix=spdk_pid137406 [2022-10-26 04:11:21.593709] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:11:21.732218] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:11:21.777567] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:11:21.778256] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 145943.30 IOPS 71.26 MiB/s
=============================================================
Total : 145943.30 IOPS 71.26 MiB/s
[2022-10-26 04:11:34.584729] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:11:34.584800] [ DPDK EAL parameters: [2022-10-26 04:11:34.584811] bdevperf [2022-10-26 04:11:34.584820] --no-shconf [2022-10-26 04:11:34.584829] -l 0 [2022-10-26 04:11:34.584837] --huge-unlink [2022-10-26 04:11:34.584846] --log-level=lib.eal:6 [2022-10-26 04:11:34.584854] --log-level=lib.cryptodev:5 [2022-10-26 04:11:34.584861] --log-level=user1:6 [2022-10-26 04:11:34.584869] --iova-mode=pa [2022-10-26 04:11:34.584877] --base-virtaddr=0x200000000000 [2022-10-26 04:11:34.584885] --match-allocations [2022-10-26 04:11:34.584893] --file-prefix=spdk_pid137431 [2022-10-26 04:11:34.584902] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:11:34.723185] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:11:34.765492] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:11:34.766168] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 151211.44 IOPS 73.83 MiB/s
=============================================================
Total : 151211.44 IOPS 73.83 MiB/s
[2022-10-26 04:11:47.573241] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:11:47.573309] [ DPDK EAL parameters: [2022-10-26 04:11:47.573320] bdevperf [2022-10-26 04:11:47.573329] --no-shconf [2022-10-26 04:11:47.573338] -l 0 [2022-10-26 04:11:47.573346] --huge-unlink [2022-10-26 04:11:47.573355] --log-level=lib.eal:6 [2022-10-26 04:11:47.573363] --log-level=lib.cryptodev:5 [2022-10-26 04:11:47.573389] --log-level=user1:6 [2022-10-26 04:11:47.573398] --iova-mode=pa [2022-10-26 04:11:47.573406] --base-virtaddr=0x200000000000 [2022-10-26 04:11:47.573414] --match-allocations [2022-10-26 04:11:47.573426] --file-prefix=spdk_pid137456 [2022-10-26 04:11:47.573434] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:11:47.711808] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:11:47.753532] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:11:47.754194] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 139449.94 IOPS 68.09 MiB/s
=============================================================
Total : 139449.94 IOPS 68.09 MiB/s
[2022-10-26 04:12:00.586747] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:00.586818] [ DPDK EAL parameters: [2022-10-26 04:12:00.586829] bdevperf [2022-10-26 04:12:00.586837] --no-shconf [2022-10-26 04:12:00.586845] -l 0 [2022-10-26 04:12:00.586853] --huge-unlink [2022-10-26 04:12:00.586860] --log-level=lib.eal:6 [2022-10-26 04:12:00.586868] --log-level=lib.cryptodev:5 [2022-10-26 04:12:00.586875] --log-level=user1:6 [2022-10-26 04:12:00.586883] --iova-mode=pa [2022-10-26 04:12:00.586890] --base-virtaddr=0x200000000000 [2022-10-26 04:12:00.586898] --match-allocations [2022-10-26 04:12:00.586905] --file-prefix=spdk_pid137481 [2022-10-26 04:12:00.586912] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:00.725651] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:00.765524] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:00.766197] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 139549.74 IOPS 68.14 MiB/s
=============================================================
Total : 139549.74 IOPS 68.14 MiB/s
[2022-10-26 04:12:13.587573] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:13.587648] [ DPDK EAL parameters: [2022-10-26 04:12:13.587659] bdevperf [2022-10-26 04:12:13.587669] --no-shconf [2022-10-26 04:12:13.587678] -l 0 [2022-10-26 04:12:13.587686] --huge-unlink [2022-10-26 04:12:13.587695] --log-level=lib.eal:6 [2022-10-26 04:12:13.587703] --log-level=lib.cryptodev:5 [2022-10-26 04:12:13.587711] --log-level=user1:6 [2022-10-26 04:12:13.587720] --iova-mode=pa [2022-10-26 04:12:13.587728] --base-virtaddr=0x200000000000 [2022-10-26 04:12:13.587736] --match-allocations [2022-10-26 04:12:13.587744] --file-prefix=spdk_pid137506 [2022-10-26 04:12:13.587752] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:13.726489] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:13.773484] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:13.774158] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 262928.44 IOPS 128.38 MiB/s
=============================================================
Total : 262928.44 IOPS 128.38 MiB/s
[2022-10-26 04:12:26.587184] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:26.587258] [ DPDK EAL parameters: [2022-10-26 04:12:26.587270] bdevperf [2022-10-26 04:12:26.587278] --no-shconf [2022-10-26 04:12:26.587287] -l 0 [2022-10-26 04:12:26.587295] --huge-unlink [2022-10-26 04:12:26.587304] --log-level=lib.eal:6 [2022-10-26 04:12:26.587312] --log-level=lib.cryptodev:5 [2022-10-26 04:12:26.587319] --log-level=user1:6 [2022-10-26 04:12:26.587328] --iova-mode=pa [2022-10-26 04:12:26.587337] --base-virtaddr=0x200000000000 [2022-10-26 04:12:26.587344] --match-allocations [2022-10-26 04:12:26.587352] --file-prefix=spdk_pid137531 [2022-10-26 04:12:26.587360] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:26.725751] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:26.765496] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:26.766167] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 242424.58 IOPS 118.37 MiB/s
=============================================================
Total : 242424.58 IOPS 118.37 MiB/s
[2022-10-26 04:12:39.856534] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:39.856602] [ DPDK EAL parameters: [2022-10-26 04:12:39.856613] bdevperf [2022-10-26 04:12:39.856621] --no-shconf [2022-10-26 04:12:39.856629] -l 0 [2022-10-26 04:12:39.856637] --huge-unlink [2022-10-26 04:12:39.856644] --log-level=lib.eal:6 [2022-10-26 04:12:39.856652] --log-level=lib.cryptodev:5 [2022-10-26 04:12:39.856659] --log-level=user1:6 [2022-10-26 04:12:39.856667] --iova-mode=pa [2022-10-26 04:12:39.856674] --base-virtaddr=0x200000000000 [2022-10-26 04:12:39.856681] --match-allocations [2022-10-26 04:12:39.856688] --file-prefix=spdk_pid137556 [2022-10-26 04:12:39.856696] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:39.994898] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:40.041502] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:40.042162] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 263549.12 IOPS 128.69 MiB/s
=============================================================
Total : 263549.12 IOPS 128.69 MiB/s
[2022-10-26 04:12:52.784402] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:52.784475] [ DPDK EAL parameters: [2022-10-26 04:12:52.784486] bdevperf [2022-10-26 04:12:52.784494] --no-shconf [2022-10-26 04:12:52.784503] -l 0 [2022-10-26 04:12:52.784511] --huge-unlink [2022-10-26 04:12:52.784520] --log-level=lib.eal:6 [2022-10-26 04:12:52.784527] --log-level=lib.cryptodev:5 [2022-10-26 04:12:52.784536] --log-level=user1:6 [2022-10-26 04:12:52.784544] --iova-mode=pa [2022-10-26 04:12:52.784552] --base-virtaddr=0x200000000000 [2022-10-26 04:12:52.784560] --match-allocations [2022-10-26 04:12:52.784568] --file-prefix=spdk_pid137581 [2022-10-26 04:12:52.784576] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:52.923433] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:52.969615] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:52.970299] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 228150.59 IOPS 111.40 MiB/s
=============================================================
Total : 228150.59 IOPS 111.40 MiB/s
[2022-10-26 04:13:05.723004] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:05.723072] [ DPDK EAL parameters: [2022-10-26 04:13:05.723082] bdevperf [2022-10-26 04:13:05.723091] --no-shconf [2022-10-26 04:13:05.723100] -l 0 [2022-10-26 04:13:05.723108] --huge-unlink [2022-10-26 04:13:05.723127] --log-level=lib.eal:6 [2022-10-26 04:13:05.723136] --log-level=lib.cryptodev:5 [2022-10-26 04:13:05.723143] --log-level=user1:6 [2022-10-26 04:13:05.723151] --iova-mode=pa [2022-10-26 04:13:05.723159] --base-virtaddr=0x200000000000 [2022-10-26 04:13:05.723167] --match-allocations [2022-10-26 04:13:05.723176] --file-prefix=spdk_pid137607 [2022-10-26 04:13:05.723184] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:05.862734] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:05.917790] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:05.918471] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 230398.16 IOPS 112.50 MiB/s
=============================================================
Total : 230398.16 IOPS 112.50 MiB/s
[2022-10-26 04:13:18.651302] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:18.651371] [ DPDK EAL parameters: [2022-10-26 04:13:18.651381] bdevperf [2022-10-26 04:13:18.651390] --no-shconf [2022-10-26 04:13:18.651399] -l 0 [2022-10-26 04:13:18.651407] --huge-unlink [2022-10-26 04:13:18.651416] --log-level=lib.eal:6 [2022-10-26 04:13:18.651423] --log-level=lib.cryptodev:5 [2022-10-26 04:13:18.651431] --log-level=user1:6 [2022-10-26 04:13:18.651440] --iova-mode=pa [2022-10-26 04:13:18.651448] --base-virtaddr=0x200000000000 [2022-10-26 04:13:18.651456] --match-allocations [2022-10-26 04:13:18.651464] --file-prefix=spdk_pid137632 [2022-10-26 04:13:18.651472] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:18.790232] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:18.829548] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:18.830238] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 229316.75 IOPS 111.97 MiB/s
=============================================================
Total : 229316.75 IOPS 111.97 MiB/s
[2022-10-26 04:13:31.605298] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:31.605366] [ DPDK EAL parameters: [2022-10-26 04:13:31.605377] bdevperf [2022-10-26 04:13:31.605386] --no-shconf [2022-10-26 04:13:31.605394] -l 0 [2022-10-26 04:13:31.605402] --huge-unlink [2022-10-26 04:13:31.605411] --log-level=lib.eal:6 [2022-10-26 04:13:31.605419] --log-level=lib.cryptodev:5 [2022-10-26 04:13:31.605426] --log-level=user1:6 [2022-10-26 04:13:31.605434] --iova-mode=pa [2022-10-26 04:13:31.605441] --base-virtaddr=0x200000000000 [2022-10-26 04:13:31.605449] --match-allocations [2022-10-26 04:13:31.605456] --file-prefix=spdk_pid137657 [2022-10-26 04:13:31.605464] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:31.743848] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:31.793535] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:31.794215] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 291240.87 IOPS 142.21 MiB/s
=============================================================
Total : 291240.87 IOPS 142.21 MiB/s
[2022-10-26 04:13:44.534674] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:44.534744] [ DPDK EAL parameters: [2022-10-26 04:13:44.534755] bdevperf [2022-10-26 04:13:44.534764] --no-shconf [2022-10-26 04:13:44.534773] -l 0 [2022-10-26 04:13:44.534780] --huge-unlink [2022-10-26 04:13:44.534789] --log-level=lib.eal:6 [2022-10-26 04:13:44.534797] --log-level=lib.cryptodev:5 [2022-10-26 04:13:44.534805] --log-level=user1:6 [2022-10-26 04:13:44.534819] --iova-mode=pa [2022-10-26 04:13:44.534827] --base-virtaddr=0x200000000000 [2022-10-26 04:13:44.534835] --match-allocations [2022-10-26 04:13:44.534842] --file-prefix=spdk_pid137682 [2022-10-26 04:13:44.534850] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:44.673093] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:44.725564] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:44.726241] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 290441.54 IOPS 141.82 MiB/s
=============================================================
Total : 290441.54 IOPS 141.82 MiB/s
[2022-10-26 04:13:57.478245] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:57.478314] [ DPDK EAL parameters: [2022-10-26 04:13:57.478324] bdevperf [2022-10-26 04:13:57.478333] --no-shconf [2022-10-26 04:13:57.478341] -l 0 [2022-10-26 04:13:57.478349] --huge-unlink [2022-10-26 04:13:57.478357] --log-level=lib.eal:6 [2022-10-26 04:13:57.478365] --log-level=lib.cryptodev:5 [2022-10-26 04:13:57.478372] --log-level=user1:6 [2022-10-26 04:13:57.478380] --iova-mode=pa [2022-10-26 04:13:57.478387] --base-virtaddr=0x200000000000 [2022-10-26 04:13:57.478395] --match-allocations [2022-10-26 04:13:57.478402] --file-prefix=spdk_pid137708 [2022-10-26 04:13:57.478409] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:57.617477] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:57.661885] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:57.662595] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 290486.27 IOPS 141.84 MiB/s
=============================================================
Total : 290486.27 IOPS 141.84 MiB/s
[2022-10-26 04:14:10.393432] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:14:10.393501] [ DPDK EAL parameters: [2022-10-26 04:14:10.393511] bdevperf [2022-10-26 04:14:10.393520] --no-shconf [2022-10-26 04:14:10.393528] -l 0 [2022-10-26 04:14:10.393536] --huge-unlink [2022-10-26 04:14:10.393544] --log-level=lib.eal:6 [2022-10-26 04:14:10.393552] --log-level=lib.cryptodev:5 [2022-10-26 04:14:10.393559] --log-level=user1:6 [2022-10-26 04:14:10.393567] --iova-mode=pa [2022-10-26 04:14:10.393575] --base-virtaddr=0x200000000000 [2022-10-26 04:14:10.393583] --match-allocations [2022-10-26 04:14:10.393590] --file-prefix=spdk_pid137735 [2022-10-26 04:14:10.393598] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:14:10.531814] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:14:10.581554] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:14:10.582236] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 285675.37 IOPS 139.49 MiB/s
=============================================================
Total : 285675.37 IOPS 139.49 MiB/s
[2022-10-26 04:14:23.317220] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:14:23.317292] [ DPDK EAL parameters: [2022-10-26 04:14:23.317303] bdevperf [2022-10-26 04:14:23.317312] --no-shconf [2022-10-26 04:14:23.317320] -l 0 [2022-10-26 04:14:23.317329] --huge-unlink [2022-10-26 04:14:23.317338] --log-level=lib.eal:6 [2022-10-26 04:14:23.317345] --log-level=lib.cryptodev:5 [2022-10-26 04:14:23.317352] --log-level=user1:6 [2022-10-26 04:14:23.317360] --iova-mode=pa [2022-10-26 04:14:23.317367] --base-virtaddr=0x200000000000 [2022-10-26 04:14:23.317375] --match-allocations [2022-10-26 04:14:23.317383] --file-prefix=spdk_pid137760 [2022-10-26 04:14:23.317391] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:14:23.456191] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:14:23.509543] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:14:23.510240] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 285787.73 IOPS 139.54 MiB/s
=============================================================
Total : 285787.73 IOPS 139.54 MiB/s
[2022-10-26 04:14:36.284758] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:14:36.284827] [ DPDK EAL parameters: [2022-10-26 04:14:36.284838] bdevperf [2022-10-26 04:14:36.284847] --no-shconf [2022-10-26 04:14:36.284855] -l 0 [2022-10-26 04:14:36.284864] --huge-unlink [2022-10-26 04:14:36.284873] --log-level=lib.eal:6 [2022-10-26 04:14:36.284880] --log-level=lib.cryptodev:5 [2022-10-26 04:14:36.284888] --log-level=user1:6 [2022-10-26 04:14:36.284896] --iova-mode=pa [2022-10-26 04:14:36.284905] --base-virtaddr=0x200000000000 [2022-10-26 04:14:36.284913] --match-allocations [2022-10-26 04:14:36.284921] --file-prefix=spdk_pid137786 [2022-10-26 04:14:36.284929] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:14:36.423316] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:14:36.465562] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:14:36.466223] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 283717.83 IOPS 138.53 MiB/s
=============================================================
Total : 283717.83 IOPS 138.53 MiB/s
[2022-10-26 04:14:49.262571] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:14:49.262640] [ DPDK EAL parameters: [2022-10-26 04:14:49.262651] bdevperf [2022-10-26 04:14:49.262661] --no-shconf [2022-10-26 04:14:49.262669] -l 0 [2022-10-26 04:14:49.262676] --huge-unlink [2022-10-26 04:14:49.262684] --log-level=lib.eal:6 [2022-10-26 04:14:49.262692] --log-level=lib.cryptodev:5 [2022-10-26 04:14:49.262699] --log-level=user1:6 [2022-10-26 04:14:49.262706] --iova-mode=pa [2022-10-26 04:14:49.262714] --base-virtaddr=0x200000000000 [2022-10-26 04:14:49.262722] --match-allocations [2022-10-26 04:14:49.262729] --file-prefix=spdk_pid137811 [2022-10-26 04:14:49.262737] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:14:49.401169] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:14:49.449515] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:14:49.450200] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 273786.89 IOPS 133.69 MiB/s
=============================================================
Total : 273786.89 IOPS 133.69 MiB/s
[2022-10-26 04:15:02.256385] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:02.256453] [ DPDK EAL parameters: [2022-10-26 04:15:02.256464] bdevperf [2022-10-26 04:15:02.256473] --no-shconf [2022-10-26 04:15:02.256481] -l 0 [2022-10-26 04:15:02.256489] --huge-unlink [2022-10-26 04:15:02.256498] --log-level=lib.eal:6 [2022-10-26 04:15:02.256506] --log-level=lib.cryptodev:5 [2022-10-26 04:15:02.256514] --log-level=user1:6 [2022-10-26 04:15:02.256522] --iova-mode=pa [2022-10-26 04:15:02.256531] --base-virtaddr=0x200000000000 [2022-10-26 04:15:02.256539] --match-allocations [2022-10-26 04:15:02.256548] --file-prefix=spdk_pid137838 [2022-10-26 04:15:02.256556] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:02.395084] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:02.453546] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:02.454235] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 273253.44 IOPS 133.42 MiB/s
=============================================================
Total : 273253.44 IOPS 133.42 MiB/s
[2022-10-26 04:15:15.273977] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:15.274047] [ DPDK EAL parameters: [2022-10-26 04:15:15.274058] bdevperf [2022-10-26 04:15:15.274067] --no-shconf [2022-10-26 04:15:15.274075] -l 0 [2022-10-26 04:15:15.274082] --huge-unlink [2022-10-26 04:15:15.274091] --log-level=lib.eal:6 [2022-10-26 04:15:15.274098] --log-level=lib.cryptodev:5 [2022-10-26 04:15:15.274105] --log-level=user1:6 [2022-10-26 04:15:15.274120] --iova-mode=pa [2022-10-26 04:15:15.274128] --base-virtaddr=0x200000000000 [2022-10-26 04:15:15.274135] --match-allocations [2022-10-26 04:15:15.274142] --file-prefix=spdk_pid137864 [2022-10-26 04:15:15.274150] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:15.412303] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:15.453532] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:15.454217] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 273765.54 IOPS 133.67 MiB/s
=============================================================
Total : 273765.54 IOPS 133.67 MiB/s
[2022-10-26 04:15:28.271983] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:28.272054] [ DPDK EAL parameters: [2022-10-26 04:15:28.272065] bdevperf [2022-10-26 04:15:28.272073] --no-shconf [2022-10-26 04:15:28.272081] -l 0 [2022-10-26 04:15:28.272089] --huge-unlink [2022-10-26 04:15:28.272098] --log-level=lib.eal:6 [2022-10-26 04:15:28.272106] --log-level=lib.cryptodev:5 [2022-10-26 04:15:28.272114] --log-level=user1:6 [2022-10-26 04:15:28.272122] --iova-mode=pa [2022-10-26 04:15:28.272130] --base-virtaddr=0x200000000000 [2022-10-26 04:15:28.272137] --match-allocations [2022-10-26 04:15:28.272145] --file-prefix=spdk_pid137889 [2022-10-26 04:15:28.272153] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:28.410856] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:28.457568] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:28.458231] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 392406.39 IOPS 191.60 MiB/s
=============================================================
Total : 392406.39 IOPS 191.60 MiB/s
[2022-10-26 04:15:41.256135] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:41.256203] [ DPDK EAL parameters: [2022-10-26 04:15:41.256213] bdevperf [2022-10-26 04:15:41.256222] --no-shconf [2022-10-26 04:15:41.256230] -l 0 [2022-10-26 04:15:41.256238] --huge-unlink [2022-10-26 04:15:41.256246] --log-level=lib.eal:6 [2022-10-26 04:15:41.256253] --log-level=lib.cryptodev:5 [2022-10-26 04:15:41.256260] --log-level=user1:6 [2022-10-26 04:15:41.256268] --iova-mode=pa [2022-10-26 04:15:41.256276] --base-virtaddr=0x200000000000 [2022-10-26 04:15:41.256283] --match-allocations [2022-10-26 04:15:41.256290] --file-prefix=spdk_pid137914 [2022-10-26 04:15:41.256298] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:41.394453] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:41.441506] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:41.442191] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 392406.89 IOPS 191.60 MiB/s
=============================================================
Total : 392406.89 IOPS 191.60 MiB/s
[2022-10-26 04:15:54.215274] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:54.215345] [ DPDK EAL parameters: [2022-10-26 04:15:54.215356] bdevperf [2022-10-26 04:15:54.215365] --no-shconf [2022-10-26 04:15:54.215374] -l 0 [2022-10-26 04:15:54.215382] --huge-unlink [2022-10-26 04:15:54.215390] --log-level=lib.eal:6 [2022-10-26 04:15:54.215398] --log-level=lib.cryptodev:5 [2022-10-26 04:15:54.215406] --log-level=user1:6 [2022-10-26 04:15:54.215414] --iova-mode=pa [2022-10-26 04:15:54.215423] --base-virtaddr=0x200000000000 [2022-10-26 04:15:54.215431] --match-allocations [2022-10-26 04:15:54.215439] --file-prefix=spdk_pid137939 [2022-10-26 04:15:54.215448] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:54.353784] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:54.405744] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:54.406405] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 392071.55 IOPS 191.44 MiB/s
=============================================================
Total : 392071.55 IOPS 191.44 MiB/s
[2022-10-26 04:16:07.163390] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:07.163460] [ DPDK EAL parameters: [2022-10-26 04:16:07.163471] bdevperf [2022-10-26 04:16:07.163480] --no-shconf [2022-10-26 04:16:07.163488] -l 0 [2022-10-26 04:16:07.163496] --huge-unlink [2022-10-26 04:16:07.163503] --log-level=lib.eal:6 [2022-10-26 04:16:07.163511] --log-level=lib.cryptodev:5 [2022-10-26 04:16:07.163518] --log-level=user1:6 [2022-10-26 04:16:07.163526] --iova-mode=pa [2022-10-26 04:16:07.163533] --base-virtaddr=0x200000000000 [2022-10-26 04:16:07.163541] --match-allocations [2022-10-26 04:16:07.163548] --file-prefix=spdk_pid137964 [2022-10-26 04:16:07.163555] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:07.302310] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:07.345541] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:07.346210] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 430379.28 IOPS 210.15 MiB/s
=============================================================
Total : 430379.28 IOPS 210.15 MiB/s
[2022-10-26 04:16:20.087240] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:20.087312] [ DPDK EAL parameters: [2022-10-26 04:16:20.087323] bdevperf [2022-10-26 04:16:20.087332] --no-shconf [2022-10-26 04:16:20.087340] -l 0 [2022-10-26 04:16:20.087348] --huge-unlink [2022-10-26 04:16:20.087357] --log-level=lib.eal:6 [2022-10-26 04:16:20.087364] --log-level=lib.cryptodev:5 [2022-10-26 04:16:20.087372] --log-level=user1:6 [2022-10-26 04:16:20.087380] --iova-mode=pa [2022-10-26 04:16:20.087388] --base-virtaddr=0x200000000000 [2022-10-26 04:16:20.087396] --match-allocations [2022-10-26 04:16:20.087404] --file-prefix=spdk_pid137989 [2022-10-26 04:16:20.087413] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:20.225818] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:20.269590] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:20.270271] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 431736.81 IOPS 210.81 MiB/s
=============================================================
Total : 431736.81 IOPS 210.81 MiB/s
[2022-10-26 04:16:33.002825] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:33.002895] [ DPDK EAL parameters: [2022-10-26 04:16:33.002905] bdevperf [2022-10-26 04:16:33.002915] --no-shconf [2022-10-26 04:16:33.002923] -l 0 [2022-10-26 04:16:33.002930] --huge-unlink [2022-10-26 04:16:33.002938] --log-level=lib.eal:6 [2022-10-26 04:16:33.002946] --log-level=lib.cryptodev:5 [2022-10-26 04:16:33.002954] --log-level=user1:6 [2022-10-26 04:16:33.002962] --iova-mode=pa [2022-10-26 04:16:33.002970] --base-virtaddr=0x200000000000 [2022-10-26 04:16:33.002977] --match-allocations [2022-10-26 04:16:33.002985] --file-prefix=spdk_pid138014 [2022-10-26 04:16:33.002992] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:33.141476] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:33.189533] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:33.190204] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 428711.29 IOPS 209.33 MiB/s
=============================================================
Total : 428711.29 IOPS 209.33 MiB/s
[2022-10-26 04:16:45.923337] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:45.923406] [ DPDK EAL parameters: [2022-10-26 04:16:45.923416] bdevperf [2022-10-26 04:16:45.923425] --no-shconf [2022-10-26 04:16:45.923433] -l 0 [2022-10-26 04:16:45.923441] --huge-unlink [2022-10-26 04:16:45.923449] --log-level=lib.eal:6 [2022-10-26 04:16:45.923456] --log-level=lib.cryptodev:5 [2022-10-26 04:16:45.923464] --log-level=user1:6 [2022-10-26 04:16:45.923471] --iova-mode=pa [2022-10-26 04:16:45.923479] --base-virtaddr=0x200000000000 [2022-10-26 04:16:45.923486] --match-allocations [2022-10-26 04:16:45.923494] --file-prefix=spdk_pid138039 [2022-10-26 04:16:45.923501] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:46.062040] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:46.101512] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:46.102194] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 485865.81 IOPS 237.24 MiB/s
=============================================================
Total : 485865.81 IOPS 237.24 MiB/s
[2022-10-26 04:16:58.841718] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:58.841789] [ DPDK EAL parameters: [2022-10-26 04:16:58.841800] bdevperf [2022-10-26 04:16:58.841809] --no-shconf [2022-10-26 04:16:58.841817] -l 0 [2022-10-26 04:16:58.841825] --huge-unlink [2022-10-26 04:16:58.841833] --log-level=lib.eal:6 [2022-10-26 04:16:58.841840] --log-level=lib.cryptodev:5 [2022-10-26 04:16:58.841848] --log-level=user1:6 [2022-10-26 04:16:58.841855] --iova-mode=pa [2022-10-26 04:16:58.841863] --base-virtaddr=0x200000000000 [2022-10-26 04:16:58.841870] --match-allocations [2022-10-26 04:16:58.841878] --file-prefix=spdk_pid138064 [2022-10-26 04:16:58.841885] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:58.980698] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:59.025507] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:59.026173] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 482781.41 IOPS 235.73 MiB/s
=============================================================
Total : 482781.41 IOPS 235.73 MiB/s
[2022-10-26 04:17:11.766933] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:17:11.767002] [ DPDK EAL parameters: [2022-10-26 04:17:11.767013] bdevperf [2022-10-26 04:17:11.767022] --no-shconf [2022-10-26 04:17:11.767030] -l 0 [2022-10-26 04:17:11.767038] --huge-unlink [2022-10-26 04:17:11.767047] --log-level=lib.eal:6 [2022-10-26 04:17:11.767055] --log-level=lib.cryptodev:5 [2022-10-26 04:17:11.767062] --log-level=user1:6 [2022-10-26 04:17:11.767071] --iova-mode=pa [2022-10-26 04:17:11.767079] --base-virtaddr=0x200000000000 [2022-10-26 04:17:11.767087] --match-allocations [2022-10-26 04:17:11.767095] --file-prefix=spdk_pid138092 [2022-10-26 04:17:11.767103] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:17:11.905315] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:17:11.945540] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:17:11.946218] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 482648.51 IOPS 235.67 MiB/s
=============================================================
Total : 482648.51 IOPS 235.67 MiB/s
[2022-10-26 04:17:24.672976] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:17:24.673047] [ DPDK EAL parameters: [2022-10-26 04:17:24.673057] bdevperf [2022-10-26 04:17:24.673066] --no-shconf [2022-10-26 04:17:24.673074] -l 0 [2022-10-26 04:17:24.673082] --huge-unlink [2022-10-26 04:17:24.673089] --log-level=lib.eal:6 [2022-10-26 04:17:24.673097] --log-level=lib.cryptodev:5 [2022-10-26 04:17:24.673104] --log-level=user1:6 [2022-10-26 04:17:24.673112] --iova-mode=pa [2022-10-26 04:17:24.673119] --base-virtaddr=0x200000000000 [2022-10-26 04:17:24.673126] --match-allocations [2022-10-26 04:17:24.673134] --file-prefix=spdk_pid138117 [2022-10-26 04:17:24.673141] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:17:24.812469] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:17:24.861970] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:17:24.862650] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 479021.22 IOPS 233.90 MiB/s
=============================================================
Total : 479021.22 IOPS 233.90 MiB/s
[2022-10-26 04:17:37.615480] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:17:37.615550] [ DPDK EAL parameters: [2022-10-26 04:17:37.615561] bdevperf [2022-10-26 04:17:37.615570] --no-shconf [2022-10-26 04:17:37.615578] -l 0 [2022-10-26 04:17:37.615586] --huge-unlink [2022-10-26 04:17:37.615594] --log-level=lib.eal:6 [2022-10-26 04:17:37.615601] --log-level=lib.cryptodev:5 [2022-10-26 04:17:37.615608] --log-level=user1:6 [2022-10-26 04:17:37.615616] --iova-mode=pa [2022-10-26 04:17:37.615623] --base-virtaddr=0x200000000000 [2022-10-26 04:17:37.615631] --match-allocations [2022-10-26 04:17:37.615639] --file-prefix=spdk_pid138142 [2022-10-26 04:17:37.615647] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:17:37.754205] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:17:37.797538] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:17:37.798220] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 479430.62 IOPS 234.10 MiB/s
=============================================================
Total : 479430.62 IOPS 234.10 MiB/s
[2022-10-26 04:17:50.615507] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:17:50.615575] [ DPDK EAL parameters: [2022-10-26 04:17:50.615586] bdevperf [2022-10-26 04:17:50.615594] --no-shconf [2022-10-26 04:17:50.615602] -l 0 [2022-10-26 04:17:50.615610] --huge-unlink [2022-10-26 04:17:50.615617] --log-level=lib.eal:6 [2022-10-26 04:17:50.615625] --log-level=lib.cryptodev:5 [2022-10-26 04:17:50.615632] --log-level=user1:6 [2022-10-26 04:17:50.615640] --iova-mode=pa [2022-10-26 04:17:50.615648] --base-virtaddr=0x200000000000 [2022-10-26 04:17:50.615655] --match-allocations [2022-10-26 04:17:50.615663] --file-prefix=spdk_pid138167 [2022-10-26 04:17:50.615670] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:17:50.752773] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:17:50.805554] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:17:50.806237] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 480153.62 IOPS 234.45 MiB/s
=============================================================
Total : 480153.62 IOPS 234.45 MiB/s
[2022-10-26 04:18:03.607723] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:03.607800] [ DPDK EAL parameters: [2022-10-26 04:18:03.607811] bdevperf [2022-10-26 04:18:03.607820] --no-shconf [2022-10-26 04:18:03.607828] -l 0 [2022-10-26 04:18:03.607836] --huge-unlink [2022-10-26 04:18:03.607844] --log-level=lib.eal:6 [2022-10-26 04:18:03.607852] --log-level=lib.cryptodev:5 [2022-10-26 04:18:03.607859] --log-level=user1:6 [2022-10-26 04:18:03.607868] --iova-mode=pa [2022-10-26 04:18:03.607876] --base-virtaddr=0x200000000000 [2022-10-26 04:18:03.607884] --match-allocations [2022-10-26 04:18:03.607892] --file-prefix=spdk_pid138192 [2022-10-26 04:18:03.607900] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:03.746385] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:03.789546] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:03.790227] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 430950.51 IOPS 210.43 MiB/s
=============================================================
Total : 430950.51 IOPS 210.43 MiB/s
[2022-10-26 04:18:16.597404] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:16.597472] [ DPDK EAL parameters: [2022-10-26 04:18:16.597483] bdevperf [2022-10-26 04:18:16.597491] --no-shconf [2022-10-26 04:18:16.597499] -l 0 [2022-10-26 04:18:16.597507] --huge-unlink [2022-10-26 04:18:16.597516] --log-level=lib.eal:6 [2022-10-26 04:18:16.597524] --log-level=lib.cryptodev:5 [2022-10-26 04:18:16.597531] --log-level=user1:6 [2022-10-26 04:18:16.597557] --iova-mode=pa [2022-10-26 04:18:16.597567] --base-virtaddr=0x200000000000 [2022-10-26 04:18:16.597575] --match-allocations [2022-10-26 04:18:16.597583] --file-prefix=spdk_pid138217 [2022-10-26 04:18:16.597599] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:16.735774] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:16.777548] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:16.778223] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 429672.51 IOPS 209.80 MiB/s
=============================================================
Total : 429672.51 IOPS 209.80 MiB/s
[2022-10-26 04:18:29.614303] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:29.614376] [ DPDK EAL parameters: [2022-10-26 04:18:29.614387] bdevperf [2022-10-26 04:18:29.614395] --no-shconf [2022-10-26 04:18:29.614403] -l 0 [2022-10-26 04:18:29.614411] --huge-unlink [2022-10-26 04:18:29.614419] --log-level=lib.eal:6 [2022-10-26 04:18:29.614426] --log-level=lib.cryptodev:5 [2022-10-26 04:18:29.614433] --log-level=user1:6 [2022-10-26 04:18:29.614441] --iova-mode=pa [2022-10-26 04:18:29.614448] --base-virtaddr=0x200000000000 [2022-10-26 04:18:29.614455] --match-allocations [2022-10-26 04:18:29.614463] --file-prefix=spdk_pid138243 [2022-10-26 04:18:29.614470] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:29.753633] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:29.801578] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:29.802241] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 430511.26 IOPS 210.21 MiB/s
=============================================================
Total : 430511.26 IOPS 210.21 MiB/s
[2022-10-26 04:18:42.616257] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:42.616325] [ DPDK EAL parameters: [2022-10-26 04:18:42.616336] bdevperf [2022-10-26 04:18:42.616345] --no-shconf [2022-10-26 04:18:42.616353] -l 0 [2022-10-26 04:18:42.616362] --huge-unlink [2022-10-26 04:18:42.616370] --log-level=lib.eal:6 [2022-10-26 04:18:42.616378] --log-level=lib.cryptodev:5 [2022-10-26 04:18:42.616386] --log-level=user1:6 [2022-10-26 04:18:42.616394] --iova-mode=pa [2022-10-26 04:18:42.616410] --base-virtaddr=0x200000000000 [2022-10-26 04:18:42.616419] --match-allocations [2022-10-26 04:18:42.616427] --file-prefix=spdk_pid138269 [2022-10-26 04:18:42.616435] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:42.754809] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:42.801477] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:42.802142] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 616823.81 IOPS 301.18 MiB/s
=============================================================
Total : 616823.81 IOPS 301.18 MiB/s
[2022-10-26 04:18:55.614634] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:55.614701] [ DPDK EAL parameters: [2022-10-26 04:18:55.614712] bdevperf [2022-10-26 04:18:55.614722] --no-shconf [2022-10-26 04:18:55.614730] -l 0 [2022-10-26 04:18:55.614737] --huge-unlink [2022-10-26 04:18:55.614745] --log-level=lib.eal:6 [2022-10-26 04:18:55.614752] --log-level=lib.cryptodev:5 [2022-10-26 04:18:55.614759] --log-level=user1:6 [2022-10-26 04:18:55.614767] --iova-mode=pa [2022-10-26 04:18:55.614774] --base-virtaddr=0x200000000000 [2022-10-26 04:18:55.614782] --match-allocations [2022-10-26 04:18:55.614789] --file-prefix=spdk_pid138294 [2022-10-26 04:18:55.614796] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:55.753382] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:55.793511] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:55.794175] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 620874.40 IOPS 303.16 MiB/s
=============================================================
Total : 620874.40 IOPS 303.16 MiB/s
[2022-10-26 04:19:08.547339] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:19:08.547409] [ DPDK EAL parameters: [2022-10-26 04:19:08.547421] bdevperf [2022-10-26 04:19:08.547430] --no-shconf [2022-10-26 04:19:08.547438] -l 0 [2022-10-26 04:19:08.547446] --huge-unlink [2022-10-26 04:19:08.547454] --log-level=lib.eal:6 [2022-10-26 04:19:08.547461] --log-level=lib.cryptodev:5 [2022-10-26 04:19:08.547469] --log-level=user1:6 [2022-10-26 04:19:08.547476] --iova-mode=pa [2022-10-26 04:19:08.547484] --base-virtaddr=0x200000000000 [2022-10-26 04:19:08.547492] --match-allocations [2022-10-26 04:19:08.547499] --file-prefix=spdk_pid138319 [2022-10-26 04:19:08.547506] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:19:08.685968] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:19:08.729521] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:19:08.730182] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 618850.10 IOPS 302.17 MiB/s
=============================================================
Total : 618850.10 IOPS 302.17 MiB/s
[2022-10-26 04:19:21.485327] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:19:21.485385] [ DPDK EAL parameters: [2022-10-26 04:19:21.485394] bdevperf [2022-10-26 04:19:21.485400] --no-shconf [2022-10-26 04:19:21.485406] -l 0 [2022-10-26 04:19:21.485411] --huge-unlink [2022-10-26 04:19:21.485418] --log-level=lib.eal:6 [2022-10-26 04:19:21.485423] --log-level=lib.cryptodev:5 [2022-10-26 04:19:21.485429] --log-level=user1:6 [2022-10-26 04:19:21.485434] --iova-mode=pa [2022-10-26 04:19:21.485440] --base-virtaddr=0x200000000000 [2022-10-26 04:19:21.485445] --match-allocations [2022-10-26 04:19:21.485451] --file-prefix=spdk_pid138345 [2022-10-26 04:19:21.485456] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:19:21.617444] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:19:21.673483] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:19:21.674157] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...
Job: Nvme0n1 (Core Mask 0x1)
Nvme0n1 : 518690.13 IOPS 253.27 MiB/s
=============================================================
Total : 518690.13 IOPS 253.27 MiB/s
[2022-10-26 04:19:34.430663] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:19:34.430733] [ DPDK EAL parameters: [2022-10-26 04:19:34.430744] bdevperf [2022-10-26 04:19:34.430753] --no-shconf [2022-10-26 04:19:34.430761] -l 0 [2022-10-26 04:19:34.430769] --huge-unlink [2022-10-26 04:19:34.430776] --log-level=lib.eal:6 [2022-10-26 04:19:34.430783] --log-level=lib.cryptodev:5 [2022-10-26 04:19:34.430790] --log-level=user1:6 [2022-10-26 04:19:34.430798] --iova-mode=pa [2022-10-26 04:19:34.430805] --base-virtaddr=0x200000000000 [2022-10-26 04:19:34.430812] --match-allocations [2022-10-26 04:19:34.430819] --file-prefix=spdk_pid138371 [2022-10-26 04:19:34.430826] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:19:34.570480] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:19:34.614037] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:19:34.614728] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
$ rm /tmp/spdk_bdev_uring_foo.conf || true
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Extract and transform fio and bdevperf output
=============================================
fio has:
* bandwidth in KiB / second
* latency is mean and in nano-seconds
* IOPS are 'raw' / base-unit
bdevperf has:
* bandwidth in MiB / second
* IOPS are 'raw' / base-unit
NOTE: The metric-context need addition of backend-options, the options are
semi-encoded by ioengine and 'label', however, that is not very precise.
{'description': '\n'
' Extract and transform fio and bdevperf output\n'
' =============================================\n'
'\n'
' fio has:\n'
'\n'
' * bandwidth in KiB / second\n'
' * latency is mean and in nano-seconds\n'
" * IOPS are 'raw' / base-unit\n"
'\n'
' bdevperf has:\n'
'\n'
' * bandwidth in MiB / second\n'
" * IOPS are 'raw' / base-unit\n"
'\n'
' NOTE: The metric-context need addition of '
'backend-options, the options are\n'
" semi-encoded by ioengine and 'label', however, that is "
'not very precise.\n',
'extras': {},
'id': '002_plot_bdevperf',
'name': 'plot_bdevperf',
'nr': 2,
'status': {'elapsed': 0.644026517868042,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'plot',
'with': {'path': 'cijoe-output',
'plot_attributes': 'plot-attributes-512.yaml',
'tool': 'bdevperf'}}
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Run fio...
{'description': '\n Run fio...\n',
'extras': {'runlog': {'cmd_01': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.output'),
'state': {'begin': 1666743838.8012276,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7809967994689941,
'end': 1666743839.5822244,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.state')},
'cmd_02': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.output'),
'state': {'begin': 1666743839.5842044,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.04334831237793,
'end': 1666743850.6275527,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.state')},
'cmd_03': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666750939,\n'
' "timestamp_ms" : '
'1666750939831,\n'
' "time" : "Wed Oct 26 04:22:19 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 298279936,\n'
' "io_kbytes" : 291289,\n'
' "bw_bytes" : 42605332,\n'
' "bw" : 41606,\n'
' "iops" : 83213.540923,\n'
' "runtime" : 7001,\n'
' "total_ios" : 582578,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2224,\n'
' "max" : 28664,\n'
' "mean" : 2408.062177,\n'
' "stddev" : 216.254296,\n'
' "N" : 582578\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 171,\n'
' "max" : 108203,\n'
' "mean" : 9002.665866,\n'
' "stddev" : '
'1272.220914,\n'
' "N" : 582578,\n'
' "percentile" : {\n'
' "1.000000" : 8640,\n'
' "5.000000" : 8640,\n'
' "10.000000" : 8768,\n'
' "20.000000" : 8768,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 8768,\n'
' "50.000000" : 8768,\n'
' "60.000000" : 8896,\n'
' "70.000000" : 8896,\n'
' "80.000000" : 8896,\n'
' "90.000000" : 9024,\n'
' "95.000000" : 9152,\n'
' "99.000000" : 15680,\n'
' "99.500000" : 16320,\n'
' "99.900000" : 19072,\n'
' "99.950000" : 21888,\n'
' "99.990000" : 55552\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8035,\n'
' "max" : 110557,\n'
' "mean" : 11410.728043,\n'
' "stddev" : '
'1289.083345,\n'
' "N" : 582578\n'
' },\n'
' "bw_min" : 41479,\n'
' "bw_max" : 41783,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'41617.071429,\n'
' "bw_dev" : 75.011245,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 82958,\n'
' "iops_max" : 83566,\n'
' "iops_mean" : '
'83234.142857,\n'
' "iops_stddev" : '
'150.022489,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.700000,\n'
' "sys_cpu" : 46.957143,\n'
' "ctx" : 582454,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.381638,\n'
' "20" : 2.541119,\n'
' "50" : 0.052868,\n'
' "100" : 0.013389,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=138675: Wed Oct 26 04:22:19 '
'2022\n'
' read: IOPS=83.2k, BW=40.6MiB/s '
'(42.6MB/s)(284MiB/7001msec)\n'
' slat (nsec): min=2224, '
'max=28664, avg=2408.06, '
'stdev=216.25\n'
' clat (nsec): min=171, '
'max=108203, avg=9002.67, '
'stdev=1272.22\n'
' lat (usec): min=8, max=110, '
'avg=11.41, stdev= 1.29\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8640], 5.00th=[ '
'8640], 10.00th=[ 8768], 20.00th=[ '
'8768],\n'
' | 30.00th=[ 8768], 40.00th=[ '
'8768], 50.00th=[ 8768], 60.00th=[ '
'8896],\n'
' | 70.00th=[ 8896], 80.00th=[ '
'8896], 90.00th=[ 9024], 95.00th=[ '
'9152],\n'
' | 99.00th=[15680], '
'99.50th=[16320], 99.90th=[19072], '
'99.95th=[21888],\n'
' | 99.99th=[55552]\n'
' bw ( KiB/s): min=41479, '
'max=41783, per=100.00%, '
'avg=41617.07, stdev=75.01, '
'samples=14\n'
' iops : min=82958, '
'max=83566, avg=83234.14, '
'stdev=150.02, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.38%, 20=2.54%, 50=0.05%, '
'100=0.01%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=11.70%, '
'sys=46.96%, ctx=582454, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=582578,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=40.6MiB/s (42.6MB/s), '
'40.6MiB/s-40.6MiB/s '
'(42.6MB/s-42.6MB/s), io=284MiB '
'(298MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.output'),
'state': {'begin': 1666743850.6302538,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7856242656707764,
'end': 1666743851.415878,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.state')},
'cmd_04': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.output'),
'state': {'begin': 1666743852.4110315,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.793879508972168,
'end': 1666743853.204911,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.state')},
'cmd_05': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.output'),
'state': {'begin': 1666743853.2070465,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.047319650650024,
'end': 1666743864.2543662,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.state')},
'cmd_06': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666750953,\n'
' "timestamp_ms" : '
'1666750953459,\n'
' "time" : "Wed Oct 26 04:22:33 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 299215872,\n'
' "io_kbytes" : 292203,\n'
' "bw_bytes" : 42739018,\n'
' "bw" : 41737,\n'
' "iops" : 83474.646479,\n'
' "runtime" : 7001,\n'
' "total_ios" : 584406,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2245,\n'
' "max" : 30237,\n'
' "mean" : 2410.540881,\n'
' "stddev" : 223.499942,\n'
' "N" : 584406\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 190,\n'
' "max" : 117380,\n'
' "mean" : 8968.711232,\n'
' "stddev" : '
'1341.037263,\n'
' "N" : 584406,\n'
' "percentile" : {\n'
' "1.000000" : 8640,\n'
' "5.000000" : 8640,\n'
' "10.000000" : 8640,\n'
' "20.000000" : 8768,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 8768,\n'
' "50.000000" : 8768,\n'
' "60.000000" : 8768,\n'
' "70.000000" : 8768,\n'
' "80.000000" : 8896,\n'
' "90.000000" : 8896,\n'
' "95.000000" : 9024,\n'
' "99.000000" : 15680,\n'
' "99.500000" : 16512,\n'
' "99.900000" : 19072,\n'
' "99.950000" : 22912,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8446,\n'
' "max" : 119784,\n'
' "mean" : 11379.252112,\n'
' "stddev" : '
'1357.748455,\n'
' "N" : 584406\n'
' },\n'
' "bw_min" : 41568,\n'
' "bw_max" : 41880,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'41747.214286,\n'
' "bw_dev" : 89.871856,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 83136,\n'
' "iops_max" : 83760,\n'
' "iops_mean" : '
'83494.500000,\n'
' "iops_stddev" : '
'179.757422,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.600000,\n'
' "sys_cpu" : 47.971429,\n'
' "ctx" : 584275,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.434147,\n'
' "20" : 2.479270,\n'
' "50" : 0.057152,\n'
' "100" : 0.018651,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=138713: Wed Oct 26 04:22:33 '
'2022\n'
' read: IOPS=83.5k, BW=40.8MiB/s '
'(42.7MB/s)(285MiB/7001msec)\n'
' slat (nsec): min=2245, '
'max=30237, avg=2410.54, '
'stdev=223.50\n'
' clat (nsec): min=190, '
'max=117380, avg=8968.71, '
'stdev=1341.04\n'
' lat (usec): min=8, max=119, '
'avg=11.38, stdev= 1.36\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8640], 5.00th=[ '
'8640], 10.00th=[ 8640], 20.00th=[ '
'8768],\n'
' | 30.00th=[ 8768], 40.00th=[ '
'8768], 50.00th=[ 8768], 60.00th=[ '
'8768],\n'
' | 70.00th=[ 8768], 80.00th=[ '
'8896], 90.00th=[ 8896], 95.00th=[ '
'9024],\n'
' | 99.00th=[15680], '
'99.50th=[16512], 99.90th=[19072], '
'99.95th=[22912],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=41568, '
'max=41880, per=100.00%, '
'avg=41747.21, stdev=89.87, '
'samples=14\n'
' iops : min=83136, '
'max=83760, avg=83494.50, '
'stdev=179.76, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 10=97.43%, '
'20=2.48%, 50=0.06%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=10.60%, '
'sys=47.97%, ctx=584275, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=584406,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=40.8MiB/s (42.7MB/s), '
'40.8MiB/s-40.8MiB/s '
'(42.7MB/s-42.7MB/s), io=285MiB '
'(299MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.output'),
'state': {'begin': 1666743864.2571094,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8469626903533936,
'end': 1666743865.104072,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.state')},
'cmd_07': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.output'),
'state': {'begin': 1666743866.22143,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.8789708614349365,
'end': 1666743867.100401,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.state')},
'cmd_08': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.output'),
'state': {'begin': 1666743867.1025667,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.065075635910034,
'end': 1666743878.1676424,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.state')},
'cmd_09': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666750967,\n'
' "timestamp_ms" : '
'1666750967372,\n'
' "time" : "Wed Oct 26 04:22:47 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 299606016,\n'
' "io_kbytes" : 292584,\n'
' "bw_bytes" : 42794745,\n'
' "bw" : 41791,\n'
' "iops" : 83583.345236,\n'
' "runtime" : 7001,\n'
' "total_ios" : 585167,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2244,\n'
' "max" : 36078,\n'
' "mean" : 2458.829481,\n'
' "stddev" : 222.580918,\n'
' "N" : 585168\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 170,\n'
' "max" : 108393,\n'
' "mean" : 8930.150774,\n'
' "stddev" : '
'1341.344004,\n'
' "N" : 585167,\n'
' "percentile" : {\n'
' "1.000000" : 8512,\n'
' "5.000000" : 8640,\n'
' "10.000000" : 8640,\n'
' "20.000000" : 8640,\n'
' "30.000000" : 8640,\n'
' "40.000000" : 8768,\n'
' "50.000000" : 8768,\n'
' "60.000000" : 8768,\n'
' "70.000000" : 8768,\n'
' "80.000000" : 8768,\n'
' "90.000000" : 8896,\n'
' "95.000000" : 9024,\n'
' "99.000000" : 15680,\n'
' "99.500000" : 16320,\n'
' "99.900000" : 19072,\n'
' "99.950000" : 22656,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8526,\n'
' "max" : 110867,\n'
' "mean" : 11388.980330,\n'
' "stddev" : '
'1357.766479,\n'
' "N" : 585167\n'
' },\n'
' "bw_min" : 41636,\n'
' "bw_max" : 41936,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'41802.000000,\n'
' "bw_dev" : 76.181968,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 83272,\n'
' "iops_max" : 83872,\n'
' "iops_mean" : '
'83604.071429,\n'
' "iops_stddev" : '
'152.343975,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.771429,\n'
' "sys_cpu" : 45.685714,\n'
' "ctx" : 585039,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.435775,\n'
' "20" : 2.483394,\n'
' "50" : 0.051097,\n'
' "100" : 0.018285,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=138751: Wed Oct 26 04:22:47 '
'2022\n'
' read: IOPS=83.6k, BW=40.8MiB/s '
'(42.8MB/s)(286MiB/7001msec)\n'
' slat (nsec): min=2244, '
'max=36078, avg=2458.83, '
'stdev=222.58\n'
' clat (nsec): min=170, '
'max=108393, avg=8930.15, '
'stdev=1341.34\n'
' lat (usec): min=8, max=110, '
'avg=11.39, stdev= 1.36\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8512], 5.00th=[ '
'8640], 10.00th=[ 8640], 20.00th=[ '
'8640],\n'
' | 30.00th=[ 8640], 40.00th=[ '
'8768], 50.00th=[ 8768], 60.00th=[ '
'8768],\n'
' | 70.00th=[ 8768], 80.00th=[ '
'8768], 90.00th=[ 8896], 95.00th=[ '
'9024],\n'
' | 99.00th=[15680], '
'99.50th=[16320], 99.90th=[19072], '
'99.95th=[22656],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=41636, '
'max=41936, per=100.00%, '
'avg=41802.00, stdev=76.18, '
'samples=14\n'
' iops : min=83272, '
'max=83872, avg=83604.07, '
'stdev=152.34, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 10=97.44%, '
'20=2.48%, 50=0.05%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=12.77%, '
'sys=45.69%, ctx=585039, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=585167,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=40.8MiB/s (42.8MB/s), '
'40.8MiB/s-40.8MiB/s '
'(42.8MB/s-42.8MB/s), io=286MiB '
'(300MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.output'),
'state': {'begin': 1666743878.1701941,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7935676574707031,
'end': 1666743878.9637618,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.state')},
'cmd_10': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.output'),
'state': {'begin': 1666743879.9947097,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7847332954406738,
'end': 1666743880.779443,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.state')},
'cmd_100': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.output'),
'state': {'begin': 1666744310.7295651,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7809097766876221,
'end': 1666744311.510475,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.state')},
'cmd_101': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.output'),
'state': {'begin': 1666744311.5130086,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.03992748260498,
'end': 1666744322.552936,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.state')},
'cmd_102': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751411,\n'
' "timestamp_ms" : '
'1666751411748,\n'
' "time" : "Wed Oct 26 04:30:11 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 737202688,\n'
' "io_kbytes" : 719924,\n'
' "bw_bytes" : 105299626,\n'
' "bw" : 102831,\n'
' "iops" : 205663.190973,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1439848,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1152,\n'
' "max" : 27401,\n'
' "mean" : 1324.465338,\n'
' "stddev" : '
'150.281294,\n'
' "N" : 1439848\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 101000,\n'
' "mean" : 8115.300295,\n'
' "stddev" : '
'2223.862930,\n'
' "N" : 1439849,\n'
' "percentile" : {\n'
' "1.000000" : 6048,\n'
' "5.000000" : 6112,\n'
' "10.000000" : 6112,\n'
' "20.000000" : 6176,\n'
' "30.000000" : 6240,\n'
' "40.000000" : 6304,\n'
' "50.000000" : 8768,\n'
' "60.000000" : 9536,\n'
' "70.000000" : 9536,\n'
' "80.000000" : 9664,\n'
' "90.000000" : 9664,\n'
' "95.000000" : 9792,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'17280,\n'
' "99.900000" : '
'18560,\n'
' "99.950000" : '
'22656,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5911,\n'
' "max" : 102202,\n'
' "mean" : 9439.765673,\n'
' "stddev" : '
'2290.846561,\n'
' "N" : 1439849\n'
' },\n'
' "bw_min" : 102294,\n'
' "bw_max" : 103753,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'102856.357143,\n'
' "bw_dev" : 489.366562,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 204588,\n'
' "iops_max" : 207506,\n'
' "iops_mean" : '
'205712.714286,\n'
' "iops_stddev" : '
'978.733124,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.900000,\n'
' "sys_cpu" : 50.014286,\n'
' "ctx" : 719850,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 96.593182,\n'
' "20" : 3.315558,\n'
' "50" : 0.057784,\n'
' "100" : 0.023961,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=140132: Wed Oct 26 04:30:11 '
'2022\n'
' read: IOPS=206k, BW=100MiB/s '
'(105MB/s)(703MiB/7001msec)\n'
' slat (nsec): min=1152, '
'max=27401, avg=1324.47, '
'stdev=150.28\n'
' clat (nsec): min=80, '
'max=101000, avg=8115.30, '
'stdev=2223.86\n'
' lat (usec): min=5, max=102, '
'avg= 9.44, stdev= 2.29\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6048], '
'5.00th=[ 6112], 10.00th=[ 6112], '
'20.00th=[ 6176],\n'
' | 30.00th=[ 6240], '
'40.00th=[ 6304], 50.00th=[ '
'8768], 60.00th=[ 9536],\n'
' | 70.00th=[ 9536], '
'80.00th=[ 9664], 90.00th=[ '
'9664], 95.00th=[ 9792],\n'
' | 99.00th=[16512], '
'99.50th=[17280], '
'99.90th=[18560], '
'99.95th=[22656],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=102294, '
'max=103753, per=100.00%, '
'avg=102856.36, stdev=489.37, '
'samples=14\n'
' iops : min=204588, '
'max=207506, avg=205712.71, '
'stdev=978.73, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=96.59%, 20=3.32%, '
'50=0.06%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=18.90%, '
'sys=50.01%, ctx=719850, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1439848,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=100MiB/s (105MB/s), '
'100MiB/s-100MiB/s '
'(105MB/s-105MB/s), io=703MiB '
'(737MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.output'),
'state': {'begin': 1666744322.5554903,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7864255905151367,
'end': 1666744323.3419158,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.state')},
'cmd_103': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.output'),
'state': {'begin': 1666744324.3314447,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7831816673278809,
'end': 1666744325.1146264,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.state')},
'cmd_104': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.output'),
'state': {'begin': 1666744325.1165314,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.035388708114624,
'end': 1666744336.15192,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.state')},
'cmd_105': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751425,\n'
' "timestamp_ms" : '
'1666751425348,\n'
' "time" : "Wed Oct 26 04:30:25 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 741398528,\n'
' "io_kbytes" : 724022,\n'
' "bw_bytes" : 105898947,\n'
' "bw" : 103416,\n'
' "iops" : 206833.738037,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1448043,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1142,\n'
' "max" : 40395,\n'
' "mean" : 1297.229181,\n'
' "stddev" : '
'146.300890,\n'
' "N" : 1448043\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 96732,\n'
' "mean" : 8087.446961,\n'
' "stddev" : '
'2216.390509,\n'
' "N" : 1448044,\n'
' "percentile" : {\n'
' "1.000000" : 5984,\n'
' "5.000000" : 6048,\n'
' "10.000000" : 6112,\n'
' "20.000000" : 6176,\n'
' "30.000000" : 6240,\n'
' "40.000000" : 6304,\n'
' "50.000000" : 8096,\n'
' "60.000000" : 9408,\n'
' "70.000000" : 9536,\n'
' "80.000000" : 9536,\n'
' "90.000000" : 9664,\n'
' "95.000000" : 9792,\n'
' "99.000000" : '
'16320,\n'
' "99.500000" : '
'17280,\n'
' "99.900000" : '
'19072,\n'
' "99.950000" : '
'26752,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6071,\n'
' "max" : 97904,\n'
' "mean" : 9384.676166,\n'
' "stddev" : '
'2292.860984,\n'
' "N" : 1448044\n'
' },\n'
' "bw_min" : 103177,\n'
' "bw_max" : 103737,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'103442.142857,\n'
' "bw_dev" : 188.649801,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 206354,\n'
' "iops_max" : 207474,\n'
' "iops_mean" : '
'206884.285714,\n'
' "iops_stddev" : '
'377.119330,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.357143,\n'
' "sys_cpu" : 51.185714,\n'
' "ctx" : 723965,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 96.716948,\n'
' "20" : 3.182640,\n'
' "50" : 0.069128,\n'
' "100" : 0.023135,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=140171: Wed Oct 26 04:30:25 '
'2022\n'
' read: IOPS=207k, BW=101MiB/s '
'(106MB/s)(707MiB/7001msec)\n'
' slat (nsec): min=1142, '
'max=40395, avg=1297.23, '
'stdev=146.30\n'
' clat (nsec): min=80, '
'max=96732, avg=8087.45, '
'stdev=2216.39\n'
' lat (nsec): min=6071, '
'max=97904, avg=9384.68, '
'stdev=2292.86\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5984], '
'5.00th=[ 6048], 10.00th=[ 6112], '
'20.00th=[ 6176],\n'
' | 30.00th=[ 6240], '
'40.00th=[ 6304], 50.00th=[ '
'8096], 60.00th=[ 9408],\n'
' | 70.00th=[ 9536], '
'80.00th=[ 9536], 90.00th=[ '
'9664], 95.00th=[ 9792],\n'
' | 99.00th=[16320], '
'99.50th=[17280], '
'99.90th=[19072], '
'99.95th=[26752],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=103177, '
'max=103737, per=100.00%, '
'avg=103442.14, stdev=188.65, '
'samples=14\n'
' iops : min=206354, '
'max=207474, avg=206884.29, '
'stdev=377.12, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=96.72%, 20=3.18%, '
'50=0.07%\n'
' lat (usec) : 100=0.02%\n'
' cpu : usr=17.36%, '
'sys=51.19%, ctx=723965, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1448043,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=101MiB/s (106MB/s), '
'101MiB/s-101MiB/s '
'(106MB/s-106MB/s), io=707MiB '
'(741MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.output'),
'state': {'begin': 1666744336.1546147,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7847998142242432,
'end': 1666744336.9394145,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.state')},
'cmd_106': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.output'),
'state': {'begin': 1666744337.9332614,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7826714515686035,
'end': 1666744338.7159328,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.state')},
'cmd_107': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.output'),
'state': {'begin': 1666744338.7180254,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.03706669807434,
'end': 1666744349.7550921,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.state')},
'cmd_108': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751438,\n'
' "timestamp_ms" : '
'1666751438950,\n'
' "time" : "Wed Oct 26 04:30:38 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 734948864,\n'
' "io_kbytes" : 717723,\n'
' "bw_bytes" : 104977698,\n'
' "bw" : 102517,\n'
' "iops" : 205034.280817,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1435445,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1182,\n'
' "max" : 27221,\n'
' "mean" : 1330.533098,\n'
' "stddev" : '
'132.496465,\n'
' "N" : 1435446\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 100,\n'
' "max" : 117019,\n'
' "mean" : 8150.587585,\n'
' "stddev" : '
'2236.609171,\n'
' "N" : 1435446,\n'
' "percentile" : {\n'
' "1.000000" : 6048,\n'
' "5.000000" : 6112,\n'
' "10.000000" : 6112,\n'
' "20.000000" : 6240,\n'
' "30.000000" : 6304,\n'
' "40.000000" : 6304,\n'
' "50.000000" : 8640,\n'
' "60.000000" : 9536,\n'
' "70.000000" : 9664,\n'
' "80.000000" : 9664,\n'
' "90.000000" : 9792,\n'
' "95.000000" : 9792,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'17280,\n'
' "99.900000" : '
'20096,\n'
' "99.950000" : '
'27008,\n'
' "99.990000" : 62208\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5871,\n'
' "max" : 118241,\n'
' "mean" : 9481.120683,\n'
' "stddev" : '
'2308.727929,\n'
' "N" : 1435446\n'
' },\n'
' "bw_min" : 102325,\n'
' "bw_max" : 102710,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'102550.214286,\n'
' "bw_dev" : 111.494173,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 204650,\n'
' "iops_max" : 205420,\n'
' "iops_mean" : '
'205100.428571,\n'
' "iops_stddev" : '
'223.226933,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.157143,\n'
' "sys_cpu" : 48.828571,\n'
' "ctx" : 717624,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 96.261577,\n'
' "20" : 3.628561,\n'
' "50" : 0.077885,\n'
' "100" : 0.021875,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=140209: Wed Oct 26 04:30:38 '
'2022\n'
' read: IOPS=205k, BW=100MiB/s '
'(105MB/s)(701MiB/7001msec)\n'
' slat (nsec): min=1182, '
'max=27221, avg=1330.53, '
'stdev=132.50\n'
' clat (nsec): min=100, '
'max=117019, avg=8150.59, '
'stdev=2236.61\n'
' lat (usec): min=5, max=118, '
'avg= 9.48, stdev= 2.31\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6048], '
'5.00th=[ 6112], 10.00th=[ 6112], '
'20.00th=[ 6240],\n'
' | 30.00th=[ 6304], '
'40.00th=[ 6304], 50.00th=[ '
'8640], 60.00th=[ 9536],\n'
' | 70.00th=[ 9664], '
'80.00th=[ 9664], 90.00th=[ '
'9792], 95.00th=[ 9792],\n'
' | 99.00th=[16512], '
'99.50th=[17280], '
'99.90th=[20096], '
'99.95th=[27008],\n'
' | 99.99th=[62208]\n'
' bw ( KiB/s): min=102325, '
'max=102710, per=100.00%, '
'avg=102550.21, stdev=111.49, '
'samples=14\n'
' iops : min=204650, '
'max=205420, avg=205100.43, '
'stdev=223.23, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=96.26%, 20=3.63%, '
'50=0.08%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=20.16%, '
'sys=48.83%, ctx=717624, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1435445,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=100MiB/s (105MB/s), '
'100MiB/s-100MiB/s '
'(105MB/s-105MB/s), io=701MiB '
'(735MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.output'),
'state': {'begin': 1666744349.7578328,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7910897731781006,
'end': 1666744350.5489225,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.state')},
'cmd_109': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.output'),
'state': {'begin': 1666744351.544084,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7850728034973145,
'end': 1666744352.3291569,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.state')},
'cmd_11': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.output'),
'state': {'begin': 1666743880.7814655,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.055826425552368,
'end': 1666743891.837292,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.state')},
'cmd_110': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.output'),
'state': {'begin': 1666744352.3311505,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.055224418640137,
'end': 1666744363.386375,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.state')},
'cmd_111': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751452,\n'
' "timestamp_ms" : '
'1666751452581,\n'
' "time" : "Wed Oct 26 04:30:52 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 726595072,\n'
' "io_kbytes" : 709565,\n'
' "bw_bytes" : 103784469,\n'
' "bw" : 101352,\n'
' "iops" : 202703.756606,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1419129,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1212,\n'
' "max" : 64080,\n'
' "mean" : 1388.671381,\n'
' "stddev" : '
'173.652155,\n'
' "N" : 1419130\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 121,\n'
' "max" : 84979,\n'
' "mean" : 8206.406104,\n'
' "stddev" : '
'2089.729687,\n'
' "N" : 1419130,\n'
' "percentile" : {\n'
' "1.000000" : 5216,\n'
' "5.000000" : 5536,\n'
' "10.000000" : 6304,\n'
' "20.000000" : 6368,\n'
' "30.000000" : 6496,\n'
' "40.000000" : 7776,\n'
' "50.000000" : 8096,\n'
' "60.000000" : 8256,\n'
' "70.000000" : 9792,\n'
' "80.000000" : 9920,\n'
' "90.000000" : '
'10048,\n'
' "95.000000" : '
'10176,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'17024,\n'
' "99.900000" : '
'18560,\n'
' "99.950000" : '
'22912,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5921,\n'
' "max" : 86271,\n'
' "mean" : 9595.077485,\n'
' "stddev" : '
'2157.718644,\n'
' "N" : 1419130\n'
' },\n'
' "bw_min" : 101000,\n'
' "bw_max" : 101608,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'101386.571429,\n'
' "bw_dev" : 192.066380,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 202000,\n'
' "iops_max" : 203216,\n'
' "iops_mean" : '
'202773.285714,\n'
' "iops_stddev" : '
'384.198953,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.300000,\n'
' "sys_cpu" : 49.800000,\n'
' "ctx" : 709604,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 87.470413,\n'
' "20" : 12.438193,\n'
' "50" : 0.058275,\n'
' "100" : 0.019730,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140247: Wed '
'Oct 26 04:30:52 2022\n'
' read: IOPS=203k, BW=99.0MiB/s '
'(104MB/s)(693MiB/7001msec)\n'
' slat (nsec): min=1212, '
'max=64080, avg=1388.67, '
'stdev=173.65\n'
' clat (nsec): min=121, '
'max=84979, avg=8206.41, '
'stdev=2089.73\n'
' lat (nsec): min=5921, '
'max=86271, avg=9595.08, '
'stdev=2157.72\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5216], '
'5.00th=[ 5536], 10.00th=[ 6304], '
'20.00th=[ 6368],\n'
' | 30.00th=[ 6496], '
'40.00th=[ 7776], 50.00th=[ '
'8096], 60.00th=[ 8256],\n'
' | 70.00th=[ 9792], '
'80.00th=[ 9920], '
'90.00th=[10048], '
'95.00th=[10176],\n'
' | 99.00th=[16512], '
'99.50th=[17024], '
'99.90th=[18560], '
'99.95th=[22912],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=101000, '
'max=101608, per=100.00%, '
'avg=101386.57, stdev=192.07, '
'samples=14\n'
' iops : min=202000, '
'max=203216, avg=202773.29, '
'stdev=384.20, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=87.47%, 20=12.44%, '
'50=0.06%\n'
' lat (usec) : 100=0.02%\n'
' cpu : usr=19.30%, '
'sys=49.80%, ctx=709604, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1419129,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=99.0MiB/s (104MB/s), '
'99.0MiB/s-99.0MiB/s '
'(104MB/s-104MB/s), io=693MiB '
'(727MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.output'),
'state': {'begin': 1666744363.389406,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7918994426727295,
'end': 1666744364.1813054,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.state')},
'cmd_112': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.output'),
'state': {'begin': 1666744365.1733491,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7847476005554199,
'end': 1666744365.9580967,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.state')},
'cmd_113': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.output'),
'state': {'begin': 1666744365.9600933,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.058468341827393,
'end': 1666744377.0185616,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.state')},
'cmd_114': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751466,\n'
' "timestamp_ms" : '
'1666751466213,\n'
' "time" : "Wed Oct 26 04:31:06 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 722791424,\n'
' "io_kbytes" : 705851,\n'
' "bw_bytes" : 103241168,\n'
' "bw" : 100821,\n'
' "iops" : 201642.765319,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1411701,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1202,\n'
' "max" : 25348,\n'
' "mean" : 1368.416727,\n'
' "stddev" : '
'146.449095,\n'
' "N" : 1411701\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 120,\n'
' "max" : 109946,\n'
' "mean" : 8258.915002,\n'
' "stddev" : '
'2093.679574,\n'
' "N" : 1411702,\n'
' "percentile" : {\n'
' "1.000000" : 5472,\n'
' "5.000000" : 6240,\n'
' "10.000000" : 6240,\n'
' "20.000000" : 6304,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 7776,\n'
' "50.000000" : 8032,\n'
' "60.000000" : 8256,\n'
' "70.000000" : 9792,\n'
' "80.000000" : 9920,\n'
' "90.000000" : 9920,\n'
' "95.000000" : '
'10048,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'17024,\n'
' "99.900000" : '
'18560,\n'
' "99.950000" : '
'23680,\n'
' "99.990000" : 60672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6092,\n'
' "max" : 111208,\n'
' "mean" : 9627.331782,\n'
' "stddev" : '
'2159.028715,\n'
' "N" : 1411702\n'
' },\n'
' "bw_min" : 100667,\n'
' "bw_max" : 101169,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'100850.000000,\n'
' "bw_dev" : 142.687394,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 201334,\n'
' "iops_max" : 202338,\n'
' "iops_mean" : '
'201700.000000,\n'
' "iops_stddev" : '
'285.374787,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 21.028571,\n'
' "sys_cpu" : 47.785714,\n'
' "ctx" : 705798,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 91.422546,\n'
' "20" : 8.482674,\n'
' "50" : 0.060565,\n'
' "100" : 0.022597,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140286: Wed '
'Oct 26 04:31:06 2022\n'
' read: IOPS=202k, BW=98.5MiB/s '
'(103MB/s)(689MiB/7001msec)\n'
' slat (nsec): min=1202, '
'max=25348, avg=1368.42, '
'stdev=146.45\n'
' clat (nsec): min=120, '
'max=109946, avg=8258.92, '
'stdev=2093.68\n'
' lat (usec): min=6, max=111, '
'avg= 9.63, stdev= 2.16\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5472], '
'5.00th=[ 6240], 10.00th=[ 6240], '
'20.00th=[ 6304],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 7776], 50.00th=[ '
'8032], 60.00th=[ 8256],\n'
' | 70.00th=[ 9792], '
'80.00th=[ 9920], 90.00th=[ '
'9920], 95.00th=[10048],\n'
' | 99.00th=[16512], '
'99.50th=[17024], '
'99.90th=[18560], '
'99.95th=[23680],\n'
' | 99.99th=[60672]\n'
' bw ( KiB/s): min=100667, '
'max=101169, per=100.00%, '
'avg=100850.00, stdev=142.69, '
'samples=14\n'
' iops : min=201334, '
'max=202338, avg=201700.00, '
'stdev=285.37, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=91.42%, 20=8.48%, '
'50=0.06%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=21.03%, '
'sys=47.79%, ctx=705798, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1411701,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=98.5MiB/s (103MB/s), '
'98.5MiB/s-98.5MiB/s '
'(103MB/s-103MB/s), io=689MiB '
'(723MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.output'),
'state': {'begin': 1666744377.02161,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7968065738677979,
'end': 1666744377.8184166,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.state')},
'cmd_115': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.output'),
'state': {'begin': 1666744378.8055432,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7906351089477539,
'end': 1666744379.5961783,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.state')},
'cmd_116': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.output'),
'state': {'begin': 1666744379.598312,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.064814329147339,
'end': 1666744390.6631262,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.state')},
'cmd_117': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751479,\n'
' "timestamp_ms" : '
'1666751479857,\n'
' "time" : "Wed Oct 26 04:31:19 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 727922688,\n'
' "io_kbytes" : 710862,\n'
' "bw_bytes" : 103974101,\n'
' "bw" : 101537,\n'
' "iops" : 203074.275104,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1421723,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1212,\n'
' "max" : 24436,\n'
' "mean" : 1384.957250,\n'
' "stddev" : '
'152.943213,\n'
' "N" : 1421723\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 120,\n'
' "max" : 77255,\n'
' "mean" : 8203.277642,\n'
' "stddev" : '
'2113.140529,\n'
' "N" : 1421724,\n'
' "percentile" : {\n'
' "1.000000" : 5280,\n'
' "5.000000" : 6176,\n'
' "10.000000" : 6240,\n'
' "20.000000" : 6304,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 7776,\n'
' "50.000000" : 8032,\n'
' "60.000000" : 8256,\n'
' "70.000000" : 9792,\n'
' "80.000000" : 9920,\n'
' "90.000000" : 9920,\n'
' "95.000000" : '
'10048,\n'
' "99.000000" : '
'16320,\n'
' "99.500000" : '
'17280,\n'
' "99.900000" : '
'18816,\n'
' "99.950000" : '
'23424,\n'
' "99.990000" : 60672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6181,\n'
' "max" : 78517,\n'
' "mean" : 9588.234953,\n'
' "stddev" : '
'2180.631735,\n'
' "N" : 1421724\n'
' },\n'
' "bw_min" : 101351,\n'
' "bw_max" : 101836,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'101561.714286,\n'
' "bw_dev" : 140.942666,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 202702,\n'
' "iops_max" : 203672,\n'
' "iops_mean" : '
'203123.428571,\n'
' "iops_stddev" : '
'281.885332,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.828571,\n'
' "sys_cpu" : 49.057143,\n'
' "ctx" : 710806,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 92.513310,\n'
' "20" : 7.385335,\n'
' "50" : 0.065695,\n'
' "100" : 0.023774,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140325: Wed '
'Oct 26 04:31:19 2022\n'
' read: IOPS=203k, BW=99.2MiB/s '
'(104MB/s)(694MiB/7001msec)\n'
' slat (nsec): min=1212, '
'max=24436, avg=1384.96, '
'stdev=152.94\n'
' clat (nsec): min=120, '
'max=77255, avg=8203.28, '
'stdev=2113.14\n'
' lat (nsec): min=6181, '
'max=78517, avg=9588.23, '
'stdev=2180.63\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5280], '
'5.00th=[ 6176], 10.00th=[ 6240], '
'20.00th=[ 6304],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 7776], 50.00th=[ '
'8032], 60.00th=[ 8256],\n'
' | 70.00th=[ 9792], '
'80.00th=[ 9920], 90.00th=[ '
'9920], 95.00th=[10048],\n'
' | 99.00th=[16320], '
'99.50th=[17280], '
'99.90th=[18816], '
'99.95th=[23424],\n'
' | 99.99th=[60672]\n'
' bw ( KiB/s): min=101351, '
'max=101836, per=100.00%, '
'avg=101561.71, stdev=140.94, '
'samples=14\n'
' iops : min=202702, '
'max=203672, avg=203123.43, '
'stdev=281.89, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=92.51%, 20=7.39%, '
'50=0.07%\n'
' lat (usec) : 100=0.02%\n'
' cpu : usr=19.83%, '
'sys=49.06%, ctx=710806, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1421723,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=99.2MiB/s (104MB/s), '
'99.2MiB/s-99.2MiB/s '
'(104MB/s-104MB/s), io=694MiB '
'(728MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.output'),
'state': {'begin': 1666744390.6660898,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7893517017364502,
'end': 1666744391.4554415,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.state')},
'cmd_118': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.output'),
'state': {'begin': 1666744393.4771495,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.782909631729126,
'end': 1666744394.260059,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.state')},
'cmd_119': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.output'),
'state': {'begin': 1666744394.2621856,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.489806890487671,
'end': 1666744405.7519925,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.state')},
'cmd_12': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666750981,\n'
' "timestamp_ms" : '
'1666750981040,\n'
' "time" : "Wed Oct 26 04:23:01 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 290464256,\n'
' "io_kbytes" : 283656,\n'
' "bw_bytes" : 41488966,\n'
' "bw" : 40516,\n'
' "iops" : 81032.995286,\n'
' "runtime" : 7001,\n'
' "total_ios" : 567312,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2304,\n'
' "max" : 40236,\n'
' "mean" : 2576.996690,\n'
' "stddev" : 232.578518,\n'
' "N" : 567313\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 340,\n'
' "max" : 70823,\n'
' "mean" : 9183.791792,\n'
' "stddev" : '
'1245.876786,\n'
' "N" : 567312,\n'
' "percentile" : {\n'
' "1.000000" : 8768,\n'
' "5.000000" : 8896,\n'
' "10.000000" : 8896,\n'
' "20.000000" : 8896,\n'
' "30.000000" : 8896,\n'
' "40.000000" : 9024,\n'
' "50.000000" : 9024,\n'
' "60.000000" : 9024,\n'
' "70.000000" : 9024,\n'
' "80.000000" : 9024,\n'
' "90.000000" : 9152,\n'
' "95.000000" : 9280,\n'
' "99.000000" : 15552,\n'
' "99.500000" : 16512,\n'
' "99.900000" : 19328,\n'
' "99.950000" : 21632,\n'
' "99.990000" : 55552\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7985,\n'
' "max" : 73398,\n'
' "mean" : 11760.788467,\n'
' "stddev" : '
'1263.053401,\n'
' "N" : 567312\n'
' },\n'
' "bw_min" : 40426,\n'
' "bw_max" : 40799,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'40525.857143,\n'
' "bw_dev" : 100.937038,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 80852,\n'
' "iops_max" : 81598,\n'
' "iops_mean" : '
'81051.714286,\n'
' "iops_stddev" : '
'201.874077,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.071429,\n'
' "sys_cpu" : 45.314286,\n'
' "ctx" : 567166,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.013220,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.317525,\n'
' "20" : 2.597512,\n'
' "50" : 0.056583,\n'
' "100" : 0.014983,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=138790: Wed '
'Oct 26 04:23:01 2022\n'
' read: IOPS=81.0k, BW=39.6MiB/s '
'(41.5MB/s)(277MiB/7001msec)\n'
' slat (nsec): min=2304, '
'max=40236, avg=2577.00, '
'stdev=232.58\n'
' clat (nsec): min=340, '
'max=70823, avg=9183.79, '
'stdev=1245.88\n'
' lat (nsec): min=7985, '
'max=73398, avg=11760.79, '
'stdev=1263.05\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8768], 5.00th=[ '
'8896], 10.00th=[ 8896], 20.00th=[ '
'8896],\n'
' | 30.00th=[ 8896], 40.00th=[ '
'9024], 50.00th=[ 9024], 60.00th=[ '
'9024],\n'
' | 70.00th=[ 9024], 80.00th=[ '
'9024], 90.00th=[ 9152], 95.00th=[ '
'9280],\n'
' | 99.00th=[15552], '
'99.50th=[16512], 99.90th=[19328], '
'99.95th=[21632],\n'
' | 99.99th=[55552]\n'
' bw ( KiB/s): min=40426, '
'max=40799, per=100.00%, '
'avg=40525.86, stdev=100.94, '
'samples=14\n'
' iops : min=80852, '
'max=81598, avg=81051.71, '
'stdev=201.87, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 10=97.32%, '
'20=2.60%, 50=0.06%, 100=0.01%\n'
' cpu : usr=14.07%, '
'sys=45.31%, ctx=567166, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=567312,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=39.6MiB/s (41.5MB/s), '
'39.6MiB/s-39.6MiB/s '
'(41.5MB/s-41.5MB/s), io=277MiB '
'(290MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.output'),
'state': {'begin': 1666743891.8404493,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7901482582092285,
'end': 1666743892.6305976,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.state')},
'cmd_120': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751494,\n'
' "timestamp_ms" : '
'1666751494875,\n'
' "time" : "Wed Oct 26 04:31:34 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 840442368,\n'
' "io_kbytes" : 820744,\n'
' "bw_bytes" : 120046045,\n'
' "bw" : 117232,\n'
' "iops" : 234464.790744,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1641488,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 230,\n'
' "max" : 20989,\n'
' "mean" : 273.106698,\n'
' "stddev" : 42.767660,\n'
' "N" : 1641488\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 270,\n'
' "max" : 112260,\n'
' "mean" : 8033.977383,\n'
' "stddev" : '
'2024.239647,\n'
' "N" : 1641489,\n'
' "percentile" : {\n'
' "1.000000" : 6368,\n'
' "5.000000" : 6432,\n'
' "10.000000" : 6496,\n'
' "20.000000" : 6944,\n'
' "30.000000" : 7392,\n'
' "40.000000" : 7456,\n'
' "50.000000" : 7648,\n'
' "60.000000" : 7840,\n'
' "70.000000" : 8640,\n'
' "80.000000" : 8768,\n'
' "90.000000" : 8896,\n'
' "95.000000" : 9024,\n'
' "99.000000" : '
'18816,\n'
' "99.500000" : '
'20096,\n'
' "99.900000" : '
'22400,\n'
' "99.950000" : '
'27776,\n'
' "99.990000" : 62208\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6081,\n'
' "max" : 112541,\n'
' "mean" : 8307.084073,\n'
' "stddev" : '
'2025.511582,\n'
' "N" : 1641489\n'
' },\n'
' "bw_min" : 116935,\n'
' "bw_max" : 117650,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'117259.714286,\n'
' "bw_dev" : 195.708350,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 233870,\n'
' "iops_max" : 235300,\n'
' "iops_mean" : '
'234519.428571,\n'
' "iops_stddev" : '
'391.416700,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 25,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.981480,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 96.879478,\n'
' "20" : 2.601055,\n'
' "50" : 0.497293,\n'
' "100" : 0.021505,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=140374: Wed Oct 26 04:31:34 '
'2022\n'
' read: IOPS=234k, BW=114MiB/s '
'(120MB/s)(802MiB/7001msec)\n'
' slat (nsec): min=230, '
'max=20989, avg=273.11, '
'stdev=42.77\n'
' clat (nsec): min=270, '
'max=112260, avg=8033.98, '
'stdev=2024.24\n'
' lat (usec): min=6, max=112, '
'avg= 8.31, stdev= 2.03\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6368], '
'5.00th=[ 6432], 10.00th=[ 6496], '
'20.00th=[ 6944],\n'
' | 30.00th=[ 7392], '
'40.00th=[ 7456], 50.00th=[ '
'7648], 60.00th=[ 7840],\n'
' | 70.00th=[ 8640], '
'80.00th=[ 8768], 90.00th=[ '
'8896], 95.00th=[ 9024],\n'
' | 99.00th=[18816], '
'99.50th=[20096], '
'99.90th=[22400], '
'99.95th=[27776],\n'
' | 99.99th=[62208]\n'
' bw ( KiB/s): min=116935, '
'max=117650, per=100.00%, '
'avg=117259.71, stdev=195.71, '
'samples=14\n'
' iops : min=233870, '
'max=235300, avg=234519.43, '
'stdev=391.42, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%, 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=96.88%, 20=2.60%, '
'50=0.50%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=25, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1641488,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=114MiB/s (120MB/s), '
'114MiB/s-114MiB/s '
'(120MB/s-120MB/s), io=802MiB '
'(840MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.output'),
'state': {'begin': 1666744405.7548473,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.794299840927124,
'end': 1666744406.5491471,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.state')},
'cmd_121': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.output'),
'state': {'begin': 1666744408.5745897,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7830724716186523,
'end': 1666744409.3576622,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.state')},
'cmd_122': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.output'),
'state': {'begin': 1666744409.3596444,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.487300872802734,
'end': 1666744420.8469453,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.state')},
'cmd_123': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751509,\n'
' "timestamp_ms" : '
'1666751509970,\n'
' "time" : "Wed Oct 26 04:31:49 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 906781696,\n'
' "io_kbytes" : 885529,\n'
' "bw_bytes" : 129521739,\n'
' "bw" : 126486,\n'
' "iops" : 252972.003999,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1771057,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 210,\n'
' "max" : 10360,\n'
' "mean" : 266.127212,\n'
' "stddev" : 24.148686,\n'
' "N" : 1771057\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 281,\n'
' "max" : 77735,\n'
' "mean" : 7425.552159,\n'
' "stddev" : '
'1737.669617,\n'
' "N" : 1771058,\n'
' "percentile" : {\n'
' "1.000000" : 6816,\n'
' "5.000000" : 7008,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7136,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7264,\n'
' "80.000000" : 7328,\n'
' "90.000000" : 7392,\n'
' "95.000000" : 7520,\n'
' "99.000000" : '
'16320,\n'
' "99.500000" : '
'20864,\n'
' "99.900000" : '
'22144,\n'
' "99.950000" : '
'22912,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5530,\n'
' "max" : 77996,\n'
' "mean" : 7691.679367,\n'
' "stddev" : '
'1738.017364,\n'
' "N" : 1771058\n'
' },\n'
' "bw_min" : 126151,\n'
' "bw_max" : 126861,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'126549.785714,\n'
' "bw_dev" : 246.186039,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 252302,\n'
' "iops_max" : 253722,\n'
' "iops_mean" : '
'253099.642857,\n'
' "iops_stddev" : '
'492.334097,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.990401,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.000831,\n'
' "20" : 2.473890,\n'
' "50" : 0.499589,\n'
' "100" : 0.025070,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=140424: Wed Oct 26 04:31:49 '
'2022\n'
' read: IOPS=253k, BW=124MiB/s '
'(130MB/s)(865MiB/7001msec)\n'
' slat (nsec): min=210, '
'max=10360, avg=266.13, '
'stdev=24.15\n'
' clat (nsec): min=281, '
'max=77735, avg=7425.55, '
'stdev=1737.67\n'
' lat (nsec): min=5530, '
'max=77996, avg=7691.68, '
'stdev=1738.02\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6816], '
'5.00th=[ 7008], 10.00th=[ 7008], '
'20.00th=[ 7072],\n'
' | 30.00th=[ 7072], '
'40.00th=[ 7136], 50.00th=[ '
'7200], 60.00th=[ 7200],\n'
' | 70.00th=[ 7264], '
'80.00th=[ 7328], 90.00th=[ '
'7392], 95.00th=[ 7520],\n'
' | 99.00th=[16320], '
'99.50th=[20864], '
'99.90th=[22144], '
'99.95th=[22912],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=126151, '
'max=126861, per=100.00%, '
'avg=126549.79, stdev=246.19, '
'samples=14\n'
' iops : min=252302, '
'max=253722, avg=253099.64, '
'stdev=492.33, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.00%, 20=2.47%, 50=0.50%, '
'100=0.03%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1771057,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=124MiB/s (130MB/s), '
'124MiB/s-124MiB/s '
'(130MB/s-130MB/s), io=865MiB '
'(907MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.output'),
'state': {'begin': 1666744420.8497894,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7872164249420166,
'end': 1666744421.6370058,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.state')},
'cmd_124': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.output'),
'state': {'begin': 1666744423.6496933,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7778506278991699,
'end': 1666744424.4275439,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.state')},
'cmd_125': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.output'),
'state': {'begin': 1666744424.4295397,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.47445011138916,
'end': 1666744435.9039898,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.state')},
'cmd_126': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751525,\n'
' "timestamp_ms" : '
'1666751525027,\n'
' "time" : "Wed Oct 26 04:32:05 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 834501120,\n'
' "io_kbytes" : 814942,\n'
' "bw_bytes" : 119197417,\n'
' "bw" : 116403,\n'
' "iops" : 232807.313241,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1629884,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 230,\n'
' "max" : 27211,\n'
' "mean" : 278.884964,\n'
' "stddev" : 29.538508,\n'
' "N" : 1629884\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 872,\n'
' "max" : 121678,\n'
' "mean" : 8094.376448,\n'
' "stddev" : '
'2042.240059,\n'
' "N" : 1629885,\n'
' "percentile" : {\n'
' "1.000000" : 6368,\n'
' "5.000000" : 6432,\n'
' "10.000000" : 6560,\n'
' "20.000000" : 7264,\n'
' "30.000000" : 7392,\n'
' "40.000000" : 7520,\n'
' "50.000000" : 7648,\n'
' "60.000000" : 7904,\n'
' "70.000000" : 8768,\n'
' "80.000000" : 8896,\n'
' "90.000000" : 8896,\n'
' "95.000000" : 9024,\n'
' "99.000000" : '
'18816,\n'
' "99.500000" : '
'20096,\n'
' "99.900000" : '
'22400,\n'
' "99.950000" : '
'28544,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5881,\n'
' "max" : 121959,\n'
' "mean" : 8373.261407,\n'
' "stddev" : '
'2043.377005,\n'
' "N" : 1629885\n'
' },\n'
' "bw_min" : 116102,\n'
' "bw_max" : 116981,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'116429.500000,\n'
' "bw_dev" : 245.284372,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 232204,\n'
' "iops_max" : 233962,\n'
' "iops_mean" : '
'232859.357143,\n'
' "iops_stddev" : '
'490.490744,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 99.990551,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.857322,\n'
' "20" : 2.646017,\n'
' "50" : 0.471567,\n'
' "100" : 0.024726,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=140474: Wed Oct 26 04:32:05 '
'2022\n'
' read: IOPS=233k, BW=114MiB/s '
'(119MB/s)(796MiB/7001msec)\n'
' slat (nsec): min=230, '
'max=27211, avg=278.88, '
'stdev=29.54\n'
' clat (nsec): min=872, '
'max=121678, avg=8094.38, '
'stdev=2042.24\n'
' lat (usec): min=5, max=121, '
'avg= 8.37, stdev= 2.04\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6368], '
'5.00th=[ 6432], 10.00th=[ 6560], '
'20.00th=[ 7264],\n'
' | 30.00th=[ 7392], '
'40.00th=[ 7520], 50.00th=[ '
'7648], 60.00th=[ 7904],\n'
' | 70.00th=[ 8768], '
'80.00th=[ 8896], 90.00th=[ '
'8896], 95.00th=[ 9024],\n'
' | 99.00th=[18816], '
'99.50th=[20096], '
'99.90th=[22400], '
'99.95th=[28544],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=116102, '
'max=116981, per=100.00%, '
'avg=116429.50, stdev=245.28, '
'samples=14\n'
' iops : min=232204, '
'max=233962, avg=232859.36, '
'stdev=490.49, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 10=96.86%, '
'20=2.65%, 50=0.47%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1629884,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=114MiB/s (119MB/s), '
'114MiB/s-114MiB/s '
'(119MB/s-119MB/s), io=796MiB '
'(835MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.output'),
'state': {'begin': 1666744435.9070897,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7856025695800781,
'end': 1666744436.6926923,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.state')},
'cmd_127': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.output'),
'state': {'begin': 1666744437.6836796,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7869439125061035,
'end': 1666744438.4706235,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.state')},
'cmd_128': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.output'),
'state': {'begin': 1666744438.4730957,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.045161724090576,
'end': 1666744449.5182574,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.state')},
'cmd_129': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751538,\n'
' "timestamp_ms" : '
'1666751538710,\n'
' "time" : "Wed Oct 26 04:32:18 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 712784384,\n'
' "io_kbytes" : 696078,\n'
' "bw_bytes" : 101811796,\n'
' "bw" : 99425,\n'
' "iops" : 198851.021283,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1392156,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1282,\n'
' "max" : 23945,\n'
' "mean" : 1414.855360,\n'
' "stddev" : '
'133.152742,\n'
' "N" : 1392156\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 100759,\n'
' "mean" : 8354.827444,\n'
' "stddev" : '
'2236.779986,\n'
' "N" : 1392157,\n'
' "percentile" : {\n'
' "1.000000" : 5536,\n'
' "5.000000" : 6240,\n'
' "10.000000" : 6304,\n'
' "20.000000" : 6368,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 6496,\n'
' "50.000000" : 8256,\n'
' "60.000000" : 9792,\n'
' "70.000000" : 9920,\n'
' "80.000000" : '
'10048,\n'
' "90.000000" : '
'10048,\n'
' "95.000000" : '
'10176,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'17024,\n'
' "99.900000" : '
'19072,\n'
' "99.950000" : '
'23168,\n'
' "99.990000" : 60160\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6342,\n'
' "max" : 102081,\n'
' "mean" : 9769.682853,\n'
' "stddev" : '
'2306.249536,\n'
' "N" : 1392157\n'
' },\n'
' "bw_min" : 99158,\n'
' "bw_max" : 99923,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'99467.928571,\n'
' "bw_dev" : 285.062062,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 198316,\n'
' "iops_max" : 199846,\n'
' "iops_mean" : '
'198936.071429,\n'
' "iops_stddev" : '
'570.072932,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.328571,\n'
' "sys_cpu" : 51.842857,\n'
' "ctx" : 696330,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 81.288448,\n'
' "20" : 18.611635,\n'
' "50" : 0.069820,\n'
' "100" : 0.020687,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2002080,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10829,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10829,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=140513: Wed '
'Oct 26 04:32:18 2022\n'
' read: IOPS=199k, BW=97.1MiB/s '
'(102MB/s)(680MiB/7001msec)\n'
' slat (nsec): min=1282, '
'max=23945, avg=1414.86, '
'stdev=133.15\n'
' clat (nsec): min=80, '
'max=100759, avg=8354.83, '
'stdev=2236.78\n'
' lat (usec): min=6, max=102, '
'avg= 9.77, stdev= 2.31\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5536], '
'5.00th=[ 6240], 10.00th=[ 6304], '
'20.00th=[ 6368],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 6496], 50.00th=[ '
'8256], 60.00th=[ 9792],\n'
' | 70.00th=[ 9920], '
'80.00th=[10048], '
'90.00th=[10048], '
'95.00th=[10176],\n'
' | 99.00th=[16512], '
'99.50th=[17024], '
'99.90th=[19072], '
'99.95th=[23168],\n'
' | 99.99th=[60160]\n'
' bw ( KiB/s): min=99158, '
'max=99923, per=100.00%, '
'avg=99467.93, stdev=285.06, '
'samples=14\n'
' iops : min=198316, '
'max=199846, avg=198936.07, '
'stdev=570.07, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=81.29%, 20=18.61%, '
'50=0.07%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=18.33%, '
'sys=51.84%, ctx=696330, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1392156,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=97.1MiB/s (102MB/s), '
'97.1MiB/s-97.1MiB/s '
'(102MB/s-102MB/s), io=680MiB '
'(713MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2002080/0, '
'merge=0/0, ticks=10829/0, '
'in_queue=10829, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.output'),
'state': {'begin': 1666744449.5212543,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8464257717132568,
'end': 1666744450.36768,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.state')},
'cmd_13': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.output'),
'state': {'begin': 1666743893.626948,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7838962078094482,
'end': 1666743894.4108443,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.state')},
'cmd_130': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.output'),
'state': {'begin': 1666744451.4471276,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.922795295715332,
'end': 1666744452.3699229,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.state')},
'cmd_131': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.output'),
'state': {'begin': 1666744452.3719704,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.149224996566772,
'end': 1666744463.5211954,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.state')},
'cmd_132': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751552,\n'
' "timestamp_ms" : '
'1666751552714,\n'
' "time" : "Wed Oct 26 04:32:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 713009152,\n'
' "io_kbytes" : 696298,\n'
' "bw_bytes" : 101843901,\n'
' "bw" : 99456,\n'
' "iops" : 198913.726610,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1392595,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1282,\n'
' "max" : 49703,\n'
' "mean" : 1410.760628,\n'
' "stddev" : '
'136.155381,\n'
' "N" : 1392595\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 102632,\n'
' "mean" : 8358.082344,\n'
' "stddev" : '
'2240.215117,\n'
' "N" : 1392596,\n'
' "percentile" : {\n'
' "1.000000" : 5536,\n'
' "5.000000" : 6304,\n'
' "10.000000" : 6304,\n'
' "20.000000" : 6368,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 6496,\n'
' "50.000000" : 8256,\n'
' "60.000000" : 9792,\n'
' "70.000000" : 9920,\n'
' "80.000000" : 9920,\n'
' "90.000000" : '
'10048,\n'
' "95.000000" : '
'10176,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'17024,\n'
' "99.900000" : '
'19072,\n'
' "99.950000" : '
'23168,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5671,\n'
' "max" : 103965,\n'
' "mean" : 9768.843017,\n'
' "stddev" : '
'2302.931642,\n'
' "N" : 1392596\n'
' },\n'
' "bw_min" : 99307,\n'
' "bw_max" : 99675,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'99497.357143,\n'
' "bw_dev" : 102.195595,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 198614,\n'
' "iops_max" : 199350,\n'
' "iops_mean" : '
'198994.785714,\n'
' "iops_stddev" : '
'204.325599,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.071429,\n'
' "sys_cpu" : 51.157143,\n'
' "ctx" : 696426,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 83.180681,\n'
' "20" : 16.725394,\n'
' "50" : 0.061612,\n'
' "100" : 0.023625,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2000153,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10851,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10851,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=140551: Wed '
'Oct 26 04:32:32 2022\n'
' read: IOPS=199k, BW=97.1MiB/s '
'(102MB/s)(680MiB/7001msec)\n'
' slat (nsec): min=1282, '
'max=49703, avg=1410.76, '
'stdev=136.16\n'
' clat (nsec): min=80, '
'max=102632, avg=8358.08, '
'stdev=2240.22\n'
' lat (usec): min=5, max=103, '
'avg= 9.77, stdev= 2.30\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5536], '
'5.00th=[ 6304], 10.00th=[ 6304], '
'20.00th=[ 6368],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 6496], 50.00th=[ '
'8256], 60.00th=[ 9792],\n'
' | 70.00th=[ 9920], '
'80.00th=[ 9920], '
'90.00th=[10048], '
'95.00th=[10176],\n'
' | 99.00th=[16512], '
'99.50th=[17024], '
'99.90th=[19072], '
'99.95th=[23168],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=99307, '
'max=99675, per=100.00%, '
'avg=99497.36, stdev=102.20, '
'samples=14\n'
' iops : min=198614, '
'max=199350, avg=198994.79, '
'stdev=204.33, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=83.18%, 20=16.73%, '
'50=0.06%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=19.07%, '
'sys=51.16%, ctx=696426, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1392595,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=97.1MiB/s (102MB/s), '
'97.1MiB/s-97.1MiB/s '
'(102MB/s-102MB/s), io=680MiB '
'(713MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2000153/0, '
'merge=0/0, ticks=10851/0, '
'in_queue=10851, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.output'),
'state': {'begin': 1666744463.523983,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 1.0828144550323486,
'end': 1666744464.6067975,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.state')},
'cmd_133': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.output'),
'state': {'begin': 1666744465.8849888,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 1.1885249614715576,
'end': 1666744467.0735137,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.state')},
'cmd_134': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.output'),
'state': {'begin': 1666744467.0755863,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.199074029922485,
'end': 1666744478.2746603,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.state')},
'cmd_135': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751567,\n'
' "timestamp_ms" : '
'1666751567468,\n'
' "time" : "Wed Oct 26 04:32:47 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 704743424,\n'
' "io_kbytes" : 688226,\n'
' "bw_bytes" : 100663251,\n'
' "bw" : 98303,\n'
' "iops" : 196607.770319,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1376451,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1302,\n'
' "max" : 115346,\n'
' "mean" : 1448.851801,\n'
' "stddev" : '
'170.761925,\n'
' "N" : 1376451\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 90,\n'
' "max" : 124113,\n'
' "mean" : 8432.095208,\n'
' "stddev" : '
'2364.959566,\n'
' "N" : 1376452,\n'
' "percentile" : {\n'
' "1.000000" : 5344,\n'
' "5.000000" : 6304,\n'
' "10.000000" : 6304,\n'
' "20.000000" : 6368,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 6496,\n'
' "50.000000" : 8896,\n'
' "60.000000" : '
'10048,\n'
' "70.000000" : '
'10048,\n'
' "80.000000" : '
'10176,\n'
' "90.000000" : '
'10176,\n'
' "95.000000" : '
'10304,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'17280,\n'
' "99.900000" : '
'23168,\n'
' "99.950000" : '
'23424,\n'
' "99.990000" : 65280\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6342,\n'
' "max" : 125696,\n'
' "mean" : 9880.947041,\n'
' "stddev" : '
'2442.322795,\n'
' "N" : 1376452\n'
' },\n'
' "bw_min" : 98131,\n'
' "bw_max" : 98526,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'98347.384615,\n'
' "bw_dev" : 105.430497,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 196262,\n'
' "iops_max" : 197052,\n'
' "iops_mean" : '
'196694.846154,\n'
' "iops_stddev" : '
'210.890116,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.671429,\n'
' "sys_cpu" : 52.142857,\n'
' "ctx" : 689064,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 59.597109,\n'
' "20" : 40.243205,\n'
' "50" : 0.122925,\n'
' "100" : 0.025064,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1979948,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10785,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10785,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=140589: Wed '
'Oct 26 04:32:47 2022\n'
' read: IOPS=197k, BW=96.0MiB/s '
'(101MB/s)(672MiB/7001msec)\n'
' slat (nsec): min=1302, '
'max=115346, avg=1448.85, '
'stdev=170.76\n'
' clat (nsec): min=90, '
'max=124113, avg=8432.10, '
'stdev=2364.96\n'
' lat (usec): min=6, max=125, '
'avg= 9.88, stdev= 2.44\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5344], '
'5.00th=[ 6304], 10.00th=[ 6304], '
'20.00th=[ 6368],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 6496], 50.00th=[ '
'8896], 60.00th=[10048],\n'
' | 70.00th=[10048], '
'80.00th=[10176], '
'90.00th=[10176], '
'95.00th=[10304],\n'
' | 99.00th=[16512], '
'99.50th=[17280], '
'99.90th=[23168], '
'99.95th=[23424],\n'
' | 99.99th=[65280]\n'
' bw ( KiB/s): min=98131, '
'max=98526, per=100.00%, '
'avg=98347.38, stdev=105.43, '
'samples=13\n'
' iops : min=196262, '
'max=197052, avg=196694.85, '
'stdev=210.89, samples=13\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=59.60%, 20=40.24%, '
'50=0.12%\n'
' lat (usec) : 100=0.03%, '
'250=0.01%\n'
' cpu : usr=18.67%, '
'sys=52.14%, ctx=689064, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1376451,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=96.0MiB/s (101MB/s), '
'96.0MiB/s-96.0MiB/s '
'(101MB/s-101MB/s), io=672MiB '
'(705MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1979948/0, '
'merge=0/0, ticks=10785/0, '
'in_queue=10785, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.output'),
'state': {'begin': 1666744478.2776206,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.796734094619751,
'end': 1666744479.0743546,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.state')},
'cmd_136': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.output'),
'state': {'begin': 1666744481.1065567,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7901074886322021,
'end': 1666744481.8966641,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.state')},
'cmd_137': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.output'),
'state': {'begin': 1666744481.898761,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.471136093139648,
'end': 1666744493.3698971,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.state')},
'cmd_138': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751582,\n'
' "timestamp_ms" : '
'1666751582489,\n'
' "time" : "Wed Oct 26 04:33:02 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 814672896,\n'
' "io_kbytes" : 795579,\n'
' "bw_bytes" : 116365218,\n'
' "bw" : 113637,\n'
' "iops" : 227275.674904,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1591157,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 17412,\n'
' "mean" : 151.050024,\n'
' "stddev" : 62.073951,\n'
' "N" : 1591157\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5881,\n'
' "max" : 120396,\n'
' "mean" : 8186.654414,\n'
' "stddev" : '
'1593.597298,\n'
' "N" : 1591157,\n'
' "percentile" : {\n'
' "1.000000" : 7328,\n'
' "5.000000" : 7648,\n'
' "10.000000" : 7712,\n'
' "20.000000" : 7776,\n'
' "30.000000" : 7840,\n'
' "40.000000" : 7904,\n'
' "50.000000" : 7968,\n'
' "60.000000" : 8032,\n'
' "70.000000" : 8096,\n'
' "80.000000" : 8160,\n'
' "90.000000" : 8256,\n'
' "95.000000" : 8384,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'19840,\n'
' "99.900000" : '
'20608,\n'
' "99.950000" : '
'21120,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6071,\n'
' "max" : 120566,\n'
' "mean" : 8337.704438,\n'
' "stddev" : '
'1596.070054,\n'
' "N" : 1591157\n'
' },\n'
' "bw_min" : 113483,\n'
' "bw_max" : 113870,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'113676.000000,\n'
' "bw_dev" : 107.724006,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 226966,\n'
' "iops_max" : 227740,\n'
' "iops_mean" : '
'227352.071429,\n'
' "iops_stddev" : '
'215.540274,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 33.528571,\n'
' "sys_cpu" : 66.442857,\n'
' "ctx" : 20,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.203818,\n'
' "2" : 50.796182,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.850279,\n'
' "20" : 2.729398,\n'
' "50" : 0.397949,\n'
' "100" : 0.022122,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=140637: Wed '
'Oct 26 04:33:02 2022\n'
' read: IOPS=227k, BW=111MiB/s '
'(116MB/s)(777MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=17412, avg=151.05, '
'stdev=62.07\n'
' clat (usec): min=5, max=120, '
'avg= 8.19, stdev= 1.59\n'
' lat (usec): min=6, max=120, '
'avg= 8.34, stdev= 1.60\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7328], '
'5.00th=[ 7648], 10.00th=[ 7712], '
'20.00th=[ 7776],\n'
' | 30.00th=[ 7840], '
'40.00th=[ 7904], 50.00th=[ '
'7968], 60.00th=[ 8032],\n'
' | 70.00th=[ 8096], '
'80.00th=[ 8160], 90.00th=[ '
'8256], 95.00th=[ 8384],\n'
' | 99.00th=[16512], '
'99.50th=[19840], '
'99.90th=[20608], '
'99.95th=[21120],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=113483, '
'max=113870, per=100.00%, '
'avg=113676.00, stdev=107.72, '
'samples=14\n'
' iops : min=226966, '
'max=227740, avg=227352.07, '
'stdev=215.54, samples=14\n'
' lat (usec) : 10=96.85%, '
'20=2.73%, 50=0.40%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=33.53%, '
'sys=66.44%, ctx=20, majf=0, '
'minf=0\n'
' IO depths : 1=49.2%, '
'2=50.8%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1591157,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=111MiB/s (116MB/s), '
'111MiB/s-111MiB/s '
'(116MB/s-116MB/s), io=777MiB '
'(815MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.output'),
'state': {'begin': 1666744493.3730302,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.787177324295044,
'end': 1666744494.1602075,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.state')},
'cmd_139': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.output'),
'state': {'begin': 1666744496.2266438,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7829077243804932,
'end': 1666744497.0095515,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.state')},
'cmd_14': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.output'),
'state': {'begin': 1666743894.412957,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.06182074546814,
'end': 1666743905.4747777,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.state')},
'cmd_140': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.output'),
'state': {'begin': 1666744497.0118256,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.449482917785645,
'end': 1666744508.4613085,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.state')},
'cmd_141': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751597,\n'
' "timestamp_ms" : '
'1666751597582,\n'
' "time" : "Wed Oct 26 04:33:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 821842432,\n'
' "io_kbytes" : 802580,\n'
' "bw_bytes" : 117389291,\n'
' "bw" : 114637,\n'
' "iops" : 229275.817740,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1605160,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 17734,\n'
' "mean" : 149.779642,\n'
' "stddev" : 53.132110,\n'
' "N" : 1605160\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5751,\n'
' "max" : 97513,\n'
' "mean" : 8131.512809,\n'
' "stddev" : '
'1581.103388,\n'
' "N" : 1605160,\n'
' "percentile" : {\n'
' "1.000000" : 7264,\n'
' "5.000000" : 7584,\n'
' "10.000000" : 7648,\n'
' "20.000000" : 7712,\n'
' "30.000000" : 7776,\n'
' "40.000000" : 7840,\n'
' "50.000000" : 7904,\n'
' "60.000000" : 7968,\n'
' "70.000000" : 8032,\n'
' "80.000000" : 8096,\n'
' "90.000000" : 8256,\n'
' "95.000000" : 8384,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'19840,\n'
' "99.900000" : '
'20864,\n'
' "99.950000" : '
'21120,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5952,\n'
' "max" : 97693,\n'
' "mean" : 8281.292451,\n'
' "stddev" : '
'1583.644423,\n'
' "N" : 1605160\n'
' },\n'
' "bw_min" : 114397,\n'
' "bw_max" : 114917,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'114668.571429,\n'
' "bw_dev" : 150.823745,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 228794,\n'
' "iops_max" : 229834,\n'
' "iops_mean" : '
'229337.214286,\n'
' "iops_stddev" : '
'301.690666,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 35.814286,\n'
' "sys_cpu" : 64.157143,\n'
' "ctx" : 21,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.234718,\n'
' "2" : 50.765282,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.891587,\n'
' "20" : 2.622293,\n'
' "50" : 0.465873,\n'
' "100" : 0.020247,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=140686: Wed '
'Oct 26 04:33:17 2022\n'
' read: IOPS=229k, BW=112MiB/s '
'(117MB/s)(784MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=17734, avg=149.78, '
'stdev=53.13\n'
' clat (nsec): min=5751, '
'max=97513, avg=8131.51, '
'stdev=1581.10\n'
' lat (nsec): min=5952, '
'max=97693, avg=8281.29, '
'stdev=1583.64\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7264], '
'5.00th=[ 7584], 10.00th=[ 7648], '
'20.00th=[ 7712],\n'
' | 30.00th=[ 7776], '
'40.00th=[ 7840], 50.00th=[ '
'7904], 60.00th=[ 7968],\n'
' | 70.00th=[ 8032], '
'80.00th=[ 8096], 90.00th=[ '
'8256], 95.00th=[ 8384],\n'
' | 99.00th=[16512], '
'99.50th=[19840], '
'99.90th=[20864], '
'99.95th=[21120],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=114397, '
'max=114917, per=100.00%, '
'avg=114668.57, stdev=150.82, '
'samples=14\n'
' iops : min=228794, '
'max=229834, avg=229337.21, '
'stdev=301.69, samples=14\n'
' lat (usec) : 10=96.89%, '
'20=2.62%, 50=0.47%, 100=0.02%\n'
' cpu : usr=35.81%, '
'sys=64.16%, ctx=21, majf=0, '
'minf=0\n'
' IO depths : 1=49.2%, '
'2=50.8%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1605160,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=112MiB/s (117MB/s), '
'112MiB/s-112MiB/s '
'(117MB/s-117MB/s), io=784MiB '
'(822MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.output'),
'state': {'begin': 1666744508.4644117,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7934420108795166,
'end': 1666744509.2578537,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.state')},
'cmd_142': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.output'),
'state': {'begin': 1666744511.3155067,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7826521396636963,
'end': 1666744512.0981588,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.state')},
'cmd_143': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.output'),
'state': {'begin': 1666744512.1005015,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.449578762054443,
'end': 1666744523.5500803,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.state')},
'cmd_144': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751612,\n'
' "timestamp_ms" : '
'1666751612671,\n'
' "time" : "Wed Oct 26 04:33:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 816369664,\n'
' "io_kbytes" : 797236,\n'
' "bw_bytes" : 116607579,\n'
' "bw" : 113874,\n'
' "iops" : 227749.035852,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1594471,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 23354,\n'
' "mean" : 150.289322,\n'
' "stddev" : 61.636434,\n'
' "N" : 1594471\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5841,\n'
' "max" : 119644,\n'
' "mean" : 8172.569133,\n'
' "stddev" : '
'1601.463886,\n'
' "N" : 1594471,\n'
' "percentile" : {\n'
' "1.000000" : 7264,\n'
' "5.000000" : 7648,\n'
' "10.000000" : 7712,\n'
' "20.000000" : 7776,\n'
' "30.000000" : 7840,\n'
' "40.000000" : 7904,\n'
' "50.000000" : 7968,\n'
' "60.000000" : 8032,\n'
' "70.000000" : 8032,\n'
' "80.000000" : 8096,\n'
' "90.000000" : 8256,\n'
' "95.000000" : 8384,\n'
' "99.000000" : '
'16512,\n'
' "99.500000" : '
'19840,\n'
' "99.900000" : '
'20864,\n'
' "99.950000" : '
'21120,\n'
' "99.990000" : 61696\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6092,\n'
' "max" : 119814,\n'
' "mean" : 8322.858455,\n'
' "stddev" : '
'1603.977324,\n'
' "N" : 1594471\n'
' },\n'
' "bw_min" : 113642,\n'
' "bw_max" : 114230,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'113906.642857,\n'
' "bw_dev" : 147.479650,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 227284,\n'
' "iops_max" : 228460,\n'
' "iops_mean" : '
'227813.357143,\n'
' "iops_stddev" : '
'295.000810,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.100000,\n'
' "sys_cpu" : 67.871429,\n'
' "ctx" : 27,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 49.172171,\n'
' "2" : 50.827829,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.864540,\n'
' "20" : 2.690359,\n'
' "50" : 0.421707,\n'
' "100" : 0.023142,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=140736: Wed '
'Oct 26 04:33:32 2022\n'
' read: IOPS=228k, BW=111MiB/s '
'(117MB/s)(779MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=23354, avg=150.29, '
'stdev=61.64\n'
' clat (usec): min=5, max=119, '
'avg= 8.17, stdev= 1.60\n'
' lat (usec): min=6, max=119, '
'avg= 8.32, stdev= 1.60\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7264], '
'5.00th=[ 7648], 10.00th=[ 7712], '
'20.00th=[ 7776],\n'
' | 30.00th=[ 7840], '
'40.00th=[ 7904], 50.00th=[ '
'7968], 60.00th=[ 8032],\n'
' | 70.00th=[ 8032], '
'80.00th=[ 8096], 90.00th=[ '
'8256], 95.00th=[ 8384],\n'
' | 99.00th=[16512], '
'99.50th=[19840], '
'99.90th=[20864], '
'99.95th=[21120],\n'
' | 99.99th=[61696]\n'
' bw ( KiB/s): min=113642, '
'max=114230, per=100.00%, '
'avg=113906.64, stdev=147.48, '
'samples=14\n'
' iops : min=227284, '
'max=228460, avg=227813.36, '
'stdev=295.00, samples=14\n'
' lat (usec) : 10=96.86%, '
'20=2.69%, 50=0.42%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=32.10%, '
'sys=67.87%, ctx=27, majf=0, '
'minf=0\n'
' IO depths : 1=49.2%, '
'2=50.8%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1594471,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=111MiB/s (117MB/s), '
'111MiB/s-111MiB/s '
'(117MB/s-117MB/s), io=779MiB '
'(816MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.output'),
'state': {'begin': 1666744523.5529957,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7888212203979492,
'end': 1666744524.341817,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.state')},
'cmd_145': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.output'),
'state': {'begin': 1666744526.4056482,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7824113368988037,
'end': 1666744527.1880596,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.state')},
'cmd_146': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.output'),
'state': {'begin': 1666744527.1904025,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.462955236434937,
'end': 1666744538.6533577,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.state')},
'cmd_147': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751627,\n'
' "timestamp_ms" : '
'1666751627772,\n'
' "time" : "Wed Oct 26 04:33:47 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 954444288,\n'
' "io_kbytes" : 932074,\n'
' "bw_bytes" : 136329708,\n'
' "bw" : 133134,\n'
' "iops" : 266268.675903,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1864147,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1552,\n'
' "max" : 23744,\n'
' "mean" : 2227.662047,\n'
' "stddev" : '
'687.257728,\n'
' "N" : 1864148\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 241,\n'
' "max" : 115497,\n'
' "mean" : 5044.965217,\n'
' "stddev" : '
'1900.368884,\n'
' "N" : 1864148,\n'
' "percentile" : {\n'
' "1.000000" : 4048,\n'
' "5.000000" : 4080,\n'
' "10.000000" : 4128,\n'
' "20.000000" : 4128,\n'
' "30.000000" : 4192,\n'
' "40.000000" : 4256,\n'
' "50.000000" : 5216,\n'
' "60.000000" : 5280,\n'
' "70.000000" : 5344,\n'
' "80.000000" : 5408,\n'
' "90.000000" : 5472,\n'
' "95.000000" : 5664,\n'
' "99.000000" : '
'14272,\n'
' "99.500000" : '
'18304,\n'
' "99.900000" : '
'22400,\n'
' "99.950000" : '
'22912,\n'
' "99.990000" : 60672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5591,\n'
' "max" : 117120,\n'
' "mean" : 7272.627264,\n'
' "stddev" : '
'1826.947438,\n'
' "N" : 1864148\n'
' },\n'
' "bw_min" : 133014,\n'
' "bw_max" : 133288,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'133165.500000,\n'
' "bw_dev" : 95.984574,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 266028,\n'
' "iops_max" : 266576,\n'
' "iops_mean" : '
'266331.000000,\n'
' "iops_stddev" : '
'191.778318,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 33.700000,\n'
' "sys_cpu" : 66.271429,\n'
' "ctx" : 25,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.022799,\n'
' "10" : 97.619233,\n'
' "20" : 1.967710,\n'
' "50" : 0.368265,\n'
' "100" : 0.019741,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140785: Wed '
'Oct 26 04:33:47 2022\n'
' read: IOPS=266k, BW=130MiB/s '
'(136MB/s)(910MiB/7001msec)\n'
' slat (nsec): min=1552, '
'max=23744, avg=2227.66, '
'stdev=687.26\n'
' clat (nsec): min=241, '
'max=115497, avg=5044.97, '
'stdev=1900.37\n'
' lat (usec): min=5, max=117, '
'avg= 7.27, stdev= 1.83\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4048], '
'5.00th=[ 4080], 10.00th=[ 4128], '
'20.00th=[ 4128],\n'
' | 30.00th=[ 4192], '
'40.00th=[ 4256], 50.00th=[ '
'5216], 60.00th=[ 5280],\n'
' | 70.00th=[ 5344], '
'80.00th=[ 5408], 90.00th=[ '
'5472], 95.00th=[ 5664],\n'
' | 99.00th=[14272], '
'99.50th=[18304], '
'99.90th=[22400], '
'99.95th=[22912],\n'
' | 99.99th=[60672]\n'
' bw ( KiB/s): min=133014, '
'max=133288, per=100.00%, '
'avg=133165.50, stdev=95.98, '
'samples=14\n'
' iops : min=266028, '
'max=266576, avg=266331.00, '
'stdev=191.78, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.02%, '
'10=97.62%, 20=1.97%, 50=0.37%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=33.70%, '
'sys=66.27%, ctx=25, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1864147,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=130MiB/s (136MB/s), '
'130MiB/s-130MiB/s '
'(136MB/s-136MB/s), io=910MiB '
'(954MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.output'),
'state': {'begin': 1666744538.6566305,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7914872169494629,
'end': 1666744539.4481177,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.state')},
'cmd_148': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.output'),
'state': {'begin': 1666744541.479785,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7818942070007324,
'end': 1666744542.2616792,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.state')},
'cmd_149': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.output'),
'state': {'begin': 1666744542.2640674,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.468403339385986,
'end': 1666744553.7324708,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.state')},
'cmd_15': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666750994,\n'
' "timestamp_ms" : '
'1666750994677,\n'
' "time" : "Wed Oct 26 04:23:14 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 285780992,\n'
' "io_kbytes" : 279083,\n'
' "bw_bytes" : 40820024,\n'
' "bw" : 39863,\n'
' "iops" : 79726.467647,\n'
' "runtime" : 7001,\n'
' "total_ios" : 558165,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2425,\n'
' "max" : 28102,\n'
' "mean" : 2654.688612,\n'
' "stddev" : 235.256413,\n'
' "N" : 558166\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 271,\n'
' "max" : 80301,\n'
' "mean" : 9230.039091,\n'
' "stddev" : '
'1260.026978,\n'
' "N" : 558165,\n'
' "percentile" : {\n'
' "1.000000" : 8896,\n'
' "5.000000" : 8896,\n'
' "10.000000" : 8896,\n'
' "20.000000" : 9024,\n'
' "30.000000" : 9024,\n'
' "40.000000" : 9024,\n'
' "50.000000" : 9024,\n'
' "60.000000" : 9024,\n'
' "70.000000" : 9152,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9152,\n'
' "95.000000" : 9408,\n'
' "99.000000" : 15296,\n'
' "99.500000" : 16064,\n'
' "99.900000" : 19328,\n'
' "99.950000" : 27520,\n'
' "99.990000" : 55552\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9147,\n'
' "max" : 82936,\n'
' "mean" : 11884.727846,\n'
' "stddev" : '
'1280.549295,\n'
' "N" : 558165\n'
' },\n'
' "bw_min" : 39774,\n'
' "bw_max" : 40040,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'39873.642857,\n'
' "bw_dev" : 66.964522,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 79548,\n'
' "iops_max" : 80080,\n'
' "iops_mean" : '
'79747.285714,\n'
' "iops_stddev" : '
'133.929045,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.842857,\n'
' "sys_cpu" : 45.942857,\n'
' "ctx" : 558046,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.336988,\n'
' "20" : 2.566087,\n'
' "50" : 0.071663,\n'
' "100" : 0.015587,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=138828: Wed '
'Oct 26 04:23:14 2022\n'
' read: IOPS=79.7k, BW=38.9MiB/s '
'(40.8MB/s)(273MiB/7001msec)\n'
' slat (nsec): min=2425, '
'max=28102, avg=2654.69, '
'stdev=235.26\n'
' clat (nsec): min=271, '
'max=80301, avg=9230.04, '
'stdev=1260.03\n'
' lat (nsec): min=9147, '
'max=82936, avg=11884.73, '
'stdev=1280.55\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8896], 5.00th=[ '
'8896], 10.00th=[ 8896], 20.00th=[ '
'9024],\n'
' | 30.00th=[ 9024], 40.00th=[ '
'9024], 50.00th=[ 9024], 60.00th=[ '
'9024],\n'
' | 70.00th=[ 9152], 80.00th=[ '
'9152], 90.00th=[ 9152], 95.00th=[ '
'9408],\n'
' | 99.00th=[15296], '
'99.50th=[16064], 99.90th=[19328], '
'99.95th=[27520],\n'
' | 99.99th=[55552]\n'
' bw ( KiB/s): min=39774, '
'max=40040, per=100.00%, '
'avg=39873.64, stdev=66.96, '
'samples=14\n'
' iops : min=79548, '
'max=80080, avg=79747.29, '
'stdev=133.93, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=97.34%, '
'20=2.57%, 50=0.07%, 100=0.02%\n'
' cpu : usr=13.84%, '
'sys=45.94%, ctx=558046, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=558165,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=38.9MiB/s (40.8MB/s), '
'38.9MiB/s-38.9MiB/s '
'(40.8MB/s-40.8MB/s), io=273MiB '
'(286MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.output'),
'state': {'begin': 1666743905.4780238,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7868468761444092,
'end': 1666743906.2648706,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.state')},
'cmd_150': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751642,\n'
' "timestamp_ms" : '
'1666751642852,\n'
' "time" : "Wed Oct 26 04:34:02 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 955027456,\n'
' "io_kbytes" : 932644,\n'
' "bw_bytes" : 136413006,\n'
' "bw" : 133215,\n'
' "iops" : 266431.366948,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1865286,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1533,\n'
' "max" : 25778,\n'
' "mean" : 2232.692276,\n'
' "stddev" : '
'689.704731,\n'
' "N" : 1865287\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 231,\n'
' "max" : 102753,\n'
' "mean" : 5046.237951,\n'
' "stddev" : '
'1921.027764,\n'
' "N" : 1865287,\n'
' "percentile" : {\n'
' "1.000000" : 4048,\n'
' "5.000000" : 4080,\n'
' "10.000000" : 4128,\n'
' "20.000000" : 4128,\n'
' "30.000000" : 4192,\n'
' "40.000000" : 4256,\n'
' "50.000000" : 5216,\n'
' "60.000000" : 5280,\n'
' "70.000000" : 5344,\n'
' "80.000000" : 5408,\n'
' "90.000000" : 5472,\n'
' "95.000000" : 5664,\n'
' "99.000000" : '
'14400,\n'
' "99.500000" : '
'18304,\n'
' "99.900000" : '
'22656,\n'
' "99.950000" : '
'22912,\n'
' "99.990000" : 61696\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5651,\n'
' "max" : 104316,\n'
' "mean" : 7278.930227,\n'
' "stddev" : '
'1850.805954,\n'
' "N" : 1865287\n'
' },\n'
' "bw_min" : 132892,\n'
' "bw_max" : 133392,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'133246.500000,\n'
' "bw_dev" : 157.042253,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 265784,\n'
' "iops_max" : 266785,\n'
' "iops_mean" : '
'266493.214286,\n'
' "iops_stddev" : '
'314.111734,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 35.300000,\n'
' "sys_cpu" : 64.671429,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.024286,\n'
' "10" : 97.589217,\n'
' "20" : 1.995404,\n'
' "50" : 0.367129,\n'
' "100" : 0.021820,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140835: Wed '
'Oct 26 04:34:02 2022\n'
' read: IOPS=266k, BW=130MiB/s '
'(136MB/s)(911MiB/7001msec)\n'
' slat (nsec): min=1533, '
'max=25778, avg=2232.69, '
'stdev=689.70\n'
' clat (nsec): min=231, '
'max=102753, avg=5046.24, '
'stdev=1921.03\n'
' lat (usec): min=5, max=104, '
'avg= 7.28, stdev= 1.85\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4048], '
'5.00th=[ 4080], 10.00th=[ 4128], '
'20.00th=[ 4128],\n'
' | 30.00th=[ 4192], '
'40.00th=[ 4256], 50.00th=[ '
'5216], 60.00th=[ 5280],\n'
' | 70.00th=[ 5344], '
'80.00th=[ 5408], 90.00th=[ '
'5472], 95.00th=[ 5664],\n'
' | 99.00th=[14400], '
'99.50th=[18304], '
'99.90th=[22656], '
'99.95th=[22912],\n'
' | 99.99th=[61696]\n'
' bw ( KiB/s): min=132892, '
'max=133392, per=100.00%, '
'avg=133246.50, stdev=157.04, '
'samples=14\n'
' iops : min=265784, '
'max=266785, avg=266493.21, '
'stdev=314.11, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.02%, '
'10=97.59%, 20=2.00%, 50=0.37%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=35.30%, '
'sys=64.67%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1865286,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=130MiB/s (136MB/s), '
'130MiB/s-130MiB/s '
'(136MB/s-136MB/s), io=911MiB '
'(955MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.output'),
'state': {'begin': 1666744553.735333,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7919230461120605,
'end': 1666744554.527256,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.state')},
'cmd_151': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.output'),
'state': {'begin': 1666744556.5428421,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7913188934326172,
'end': 1666744557.334161,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.state')},
'cmd_152': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.output'),
'state': {'begin': 1666744557.3363864,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.450576782226562,
'end': 1666744568.7869632,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.state')},
'cmd_153': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751657,\n'
' "timestamp_ms" : '
'1666751657907,\n'
' "time" : "Wed Oct 26 04:34:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 949441536,\n'
' "io_kbytes" : 927189,\n'
' "bw_bytes" : 135615131,\n'
' "bw" : 132436,\n'
' "iops" : 264873.018140,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1854376,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1552,\n'
' "max" : 24926,\n'
' "mean" : 2245.204077,\n'
' "stddev" : '
'687.051828,\n'
' "N" : 1854377\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 230,\n'
' "max" : 103073,\n'
' "mean" : 5066.601525,\n'
' "stddev" : '
'1903.677678,\n'
' "N" : 1854377,\n'
' "percentile" : {\n'
' "1.000000" : 4080,\n'
' "5.000000" : 4128,\n'
' "10.000000" : 4128,\n'
' "20.000000" : 4192,\n'
' "30.000000" : 4192,\n'
' "40.000000" : 4256,\n'
' "50.000000" : 5216,\n'
' "60.000000" : 5280,\n'
' "70.000000" : 5408,\n'
' "80.000000" : 5472,\n'
' "90.000000" : 5536,\n'
' "95.000000" : 5664,\n'
' "99.000000" : '
'14400,\n'
' "99.500000" : '
'18304,\n'
' "99.900000" : '
'22400,\n'
' "99.950000" : '
'22912,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5641,\n'
' "max" : 104686,\n'
' "mean" : 7311.805603,\n'
' "stddev" : '
'1830.883007,\n'
' "N" : 1854377\n'
' },\n'
' "bw_min" : 132278,\n'
' "bw_max" : 133014,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'132478.384615,\n'
' "bw_dev" : 208.302880,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 264556,\n'
' "iops_max" : 266028,\n'
' "iops_mean" : '
'264957.000000,\n'
' "iops_stddev" : '
'416.682533,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.785714,\n'
' "sys_cpu" : 69.185714,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.017310,\n'
' "10" : 97.592667,\n'
' "20" : 1.983039,\n'
' "50" : 0.384172,\n'
' "100" : 0.020600,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140884: Wed '
'Oct 26 04:34:17 2022\n'
' read: IOPS=265k, BW=129MiB/s '
'(136MB/s)(905MiB/7001msec)\n'
' slat (nsec): min=1552, '
'max=24926, avg=2245.20, '
'stdev=687.05\n'
' clat (nsec): min=230, '
'max=103073, avg=5066.60, '
'stdev=1903.68\n'
' lat (usec): min=5, max=104, '
'avg= 7.31, stdev= 1.83\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4080], '
'5.00th=[ 4128], 10.00th=[ 4128], '
'20.00th=[ 4192],\n'
' | 30.00th=[ 4192], '
'40.00th=[ 4256], 50.00th=[ '
'5216], 60.00th=[ 5280],\n'
' | 70.00th=[ 5408], '
'80.00th=[ 5472], 90.00th=[ '
'5536], 95.00th=[ 5664],\n'
' | 99.00th=[14400], '
'99.50th=[18304], '
'99.90th=[22400], '
'99.95th=[22912],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=132278, '
'max=133014, per=100.00%, '
'avg=132478.38, stdev=208.30, '
'samples=13\n'
' iops : min=264556, '
'max=266028, avg=264957.00, '
'stdev=416.68, samples=13\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.02%, '
'10=97.59%, 20=1.98%, 50=0.38%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=30.79%, '
'sys=69.19%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1854376,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=129MiB/s (136MB/s), '
'129MiB/s-129MiB/s '
'(136MB/s-136MB/s), io=905MiB '
'(949MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.output'),
'state': {'begin': 1666744568.7901118,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8964223861694336,
'end': 1666744569.6865342,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.state')},
'cmd_154': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.output'),
'state': {'begin': 1666744570.6834624,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7792384624481201,
'end': 1666744571.4627008,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.state')},
'cmd_155': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.output'),
'state': {'begin': 1666744571.464903,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.05773639678955,
'end': 1666744582.5226395,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.state')},
'cmd_156': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751671,\n'
' "timestamp_ms" : '
'1666751671714,\n'
' "time" : "Wed Oct 26 04:34:31 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 764834304,\n'
' "io_kbytes" : 746908,\n'
' "bw_bytes" : 109246436,\n'
' "bw" : 106685,\n'
' "iops" : 213371.804028,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1493816,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1382,\n'
' "max" : 25729,\n'
' "mean" : 1532.744973,\n'
' "stddev" : '
'144.972706,\n'
' "N" : 1493816\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 130,\n'
' "max" : 117480,\n'
' "mean" : 7572.991351,\n'
' "stddev" : '
'2518.779534,\n'
' "N" : 1493817,\n'
' "percentile" : {\n'
' "1.000000" : 5152,\n'
' "5.000000" : 5152,\n'
' "10.000000" : 5216,\n'
' "20.000000" : 5280,\n'
' "30.000000" : 5344,\n'
' "40.000000" : 5472,\n'
' "50.000000" : 9024,\n'
' "60.000000" : 9024,\n'
' "70.000000" : 9152,\n'
' "80.000000" : 9280,\n'
' "90.000000" : 9408,\n'
' "95.000000" : '
'10432,\n'
' "99.000000" : '
'17024,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'19840,\n'
' "99.950000" : '
'23168,\n'
' "99.990000" : 60160\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6092,\n'
' "max" : 119193,\n'
' "mean" : 9105.736391,\n'
' "stddev" : '
'2595.087936,\n'
' "N" : 1493817\n'
' },\n'
' "bw_min" : 106453,\n'
' "bw_max" : 106968,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106711.785714,\n'
' "bw_dev" : 136.440000,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 212906,\n'
' "iops_max" : 213936,\n'
' "iops_mean" : '
'213423.428571,\n'
' "iops_stddev" : '
'272.960328,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.857143,\n'
' "sys_cpu" : 57.628571,\n'
' "ctx" : 747066,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 94.517866,\n'
' "20" : 5.375160,\n'
' "50" : 0.077787,\n'
' "100" : 0.020351,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140924: Wed '
'Oct 26 04:34:31 2022\n'
' read: IOPS=213k, BW=104MiB/s '
'(109MB/s)(729MiB/7001msec)\n'
' slat (nsec): min=1382, '
'max=25729, avg=1532.74, '
'stdev=144.97\n'
' clat (nsec): min=130, '
'max=117480, avg=7572.99, '
'stdev=2518.78\n'
' lat (usec): min=6, max=119, '
'avg= 9.11, stdev= 2.60\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5152], '
'5.00th=[ 5152], 10.00th=[ 5216], '
'20.00th=[ 5280],\n'
' | 30.00th=[ 5344], '
'40.00th=[ 5472], 50.00th=[ '
'9024], 60.00th=[ 9024],\n'
' | 70.00th=[ 9152], '
'80.00th=[ 9280], 90.00th=[ '
'9408], 95.00th=[10432],\n'
' | 99.00th=[17024], '
'99.50th=[18048], '
'99.90th=[19840], '
'99.95th=[23168],\n'
' | 99.99th=[60160]\n'
' bw ( KiB/s): min=106453, '
'max=106968, per=100.00%, '
'avg=106711.79, stdev=136.44, '
'samples=14\n'
' iops : min=212906, '
'max=213936, avg=213423.43, '
'stdev=272.96, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=94.52%, 20=5.38%, 50=0.08%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=19.86%, '
'sys=57.63%, ctx=747066, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1493816,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=729MiB '
'(765MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.output'),
'state': {'begin': 1666744582.525979,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7881765365600586,
'end': 1666744583.3141556,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.state')},
'cmd_157': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.output'),
'state': {'begin': 1666744584.3125513,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7895863056182861,
'end': 1666744585.1021376,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.state')},
'cmd_158': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.output'),
'state': {'begin': 1666744585.1042175,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.063555002212524,
'end': 1666744596.1677725,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.state')},
'cmd_159': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751685,\n'
' "timestamp_ms" : '
'1666751685358,\n'
' "time" : "Wed Oct 26 04:34:45 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 765723648,\n'
' "io_kbytes" : 747777,\n'
' "bw_bytes" : 109373467,\n'
' "bw" : 106810,\n'
' "iops" : 213619.911441,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1495553,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1352,\n'
' "max" : 22452,\n'
' "mean" : 1493.274781,\n'
' "stddev" : '
'148.492491,\n'
' "N" : 1495553\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 130,\n'
' "max" : 122009,\n'
' "mean" : 7620.069698,\n'
' "stddev" : '
'2548.142265,\n'
' "N" : 1495554,\n'
' "percentile" : {\n'
' "1.000000" : 5152,\n'
' "5.000000" : 5216,\n'
' "10.000000" : 5216,\n'
' "20.000000" : 5344,\n'
' "30.000000" : 5408,\n'
' "40.000000" : 5536,\n'
' "50.000000" : 8896,\n'
' "60.000000" : 9024,\n'
' "70.000000" : 9024,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9408,\n'
' "95.000000" : '
'10560,\n'
' "99.000000" : '
'17280,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'20608,\n'
' "99.950000" : '
'23168,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6161,\n'
' "max" : 123401,\n'
' "mean" : 9113.344539,\n'
' "stddev" : '
'2616.635881,\n'
' "N" : 1495554\n'
' },\n'
' "bw_min" : 106727,\n'
' "bw_max" : 106938,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106835.214286,\n'
' "bw_dev" : 72.031167,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 213454,\n'
' "iops_max" : 213876,\n'
' "iops_mean" : '
'213670.500000,\n'
' "iops_stddev" : '
'144.097590,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.214286,\n'
' "sys_cpu" : 60.571429,\n'
' "ctx" : 748054,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 91.554763,\n'
' "20" : 8.329695,\n'
' "50" : 0.084517,\n'
' "100" : 0.016984,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140964: Wed '
'Oct 26 04:34:45 2022\n'
' read: IOPS=214k, BW=104MiB/s '
'(109MB/s)(730MiB/7001msec)\n'
' slat (nsec): min=1352, '
'max=22452, avg=1493.27, '
'stdev=148.49\n'
' clat (nsec): min=130, '
'max=122009, avg=7620.07, '
'stdev=2548.14\n'
' lat (usec): min=6, max=123, '
'avg= 9.11, stdev= 2.62\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5152], '
'5.00th=[ 5216], 10.00th=[ 5216], '
'20.00th=[ 5344],\n'
' | 30.00th=[ 5408], '
'40.00th=[ 5536], 50.00th=[ '
'8896], 60.00th=[ 9024],\n'
' | 70.00th=[ 9024], '
'80.00th=[ 9152], 90.00th=[ '
'9408], 95.00th=[10560],\n'
' | 99.00th=[17280], '
'99.50th=[18048], '
'99.90th=[20608], '
'99.95th=[23168],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=106727, '
'max=106938, per=100.00%, '
'avg=106835.21, stdev=72.03, '
'samples=14\n'
' iops : min=213454, '
'max=213876, avg=213670.50, '
'stdev=144.10, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=91.55%, 20=8.33%, 50=0.08%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=16.21%, '
'sys=60.57%, ctx=748054, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1495553,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=730MiB '
'(766MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.output'),
'state': {'begin': 1666744596.170988,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7937331199645996,
'end': 1666744596.9647212,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.state')},
'cmd_16': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.output'),
'state': {'begin': 1666743907.2699497,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7967319488525391,
'end': 1666743908.0666816,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.state')},
'cmd_160': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.output'),
'state': {'begin': 1666744597.9525113,
'cmd': 'rm '
'/tmp/fio-output_512_2_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7826642990112305,
'end': 1666744598.7351756,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.state')},
'cmd_161': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.output'),
'state': {'begin': 1666744598.7371712,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_io_uring-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.070218324661255,
'end': 1666744609.8073895,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.state')},
'cmd_162': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751698,\n'
' "timestamp_ms" : '
'1666751698998,\n'
' "time" : "Wed Oct 26 04:34:58 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 762477568,\n'
' "io_kbytes" : 744607,\n'
' "bw_bytes" : 108909808,\n'
' "bw" : 106357,\n'
' "iops" : 212714.326525,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1489213,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1352,\n'
' "max" : 26169,\n'
' "mean" : 1505.332639,\n'
' "stddev" : '
'162.198529,\n'
' "N" : 1489213\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 140,\n'
' "max" : 121859,\n'
' "mean" : 7639.105499,\n'
' "stddev" : '
'2552.195648,\n'
' "N" : 1489214,\n'
' "percentile" : {\n'
' "1.000000" : 5088,\n'
' "5.000000" : 5152,\n'
' "10.000000" : 5216,\n'
' "20.000000" : 5280,\n'
' "30.000000" : 5344,\n'
' "40.000000" : 5600,\n'
' "50.000000" : 8896,\n'
' "60.000000" : 9024,\n'
' "70.000000" : 9024,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9664,\n'
' "95.000000" : '
'10560,\n'
' "99.000000" : '
'17280,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'20608,\n'
' "99.950000" : '
'23168,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6061,\n'
' "max" : 123251,\n'
' "mean" : 9144.438204,\n'
' "stddev" : '
'2629.067758,\n'
' "N" : 1489214\n'
' },\n'
' "bw_min" : 105852,\n'
' "bw_max" : 106636,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'106381.928571,\n'
' "bw_dev" : 196.151615,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 211705,\n'
' "iops_max" : 213272,\n'
' "iops_mean" : '
'212763.928571,\n'
' "iops_stddev" : '
'392.095448,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.128571,\n'
' "sys_cpu" : 59.557143,\n'
' "ctx" : 745014,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 90.268484,\n'
' "20" : 9.615884,\n'
' "50" : 0.085481,\n'
' "100" : 0.017727,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141004: Wed '
'Oct 26 04:34:58 2022\n'
' read: IOPS=213k, BW=104MiB/s '
'(109MB/s)(727MiB/7001msec)\n'
' slat (nsec): min=1352, '
'max=26169, avg=1505.33, '
'stdev=162.20\n'
' clat (nsec): min=140, '
'max=121859, avg=7639.11, '
'stdev=2552.20\n'
' lat (usec): min=6, max=123, '
'avg= 9.14, stdev= 2.63\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5088], '
'5.00th=[ 5152], 10.00th=[ 5216], '
'20.00th=[ 5280],\n'
' | 30.00th=[ 5344], '
'40.00th=[ 5600], 50.00th=[ '
'8896], 60.00th=[ 9024],\n'
' | 70.00th=[ 9024], '
'80.00th=[ 9152], 90.00th=[ '
'9664], 95.00th=[10560],\n'
' | 99.00th=[17280], '
'99.50th=[18048], '
'99.90th=[20608], '
'99.95th=[23168],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=105852, '
'max=106636, per=100.00%, '
'avg=106381.93, stdev=196.15, '
'samples=14\n'
' iops : min=211705, '
'max=213272, avg=212763.93, '
'stdev=392.10, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=90.27%, 20=9.62%, 50=0.09%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=17.13%, '
'sys=59.56%, ctx=745014, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1489213,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=104MiB/s (109MB/s), '
'104MiB/s-104MiB/s '
'(109MB/s-109MB/s), io=727MiB '
'(762MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.output'),
'state': {'begin': 1666744609.8105936,
'cmd': 'cat '
'/tmp/fio-output_512_2_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7891902923583984,
'end': 1666744610.599784,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.state')},
'cmd_163': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.output'),
'state': {'begin': 1666744611.586182,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7797496318817139,
'end': 1666744612.3659317,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.state')},
'cmd_164': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.output'),
'state': {'begin': 1666744612.367877,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.070231676101685,
'end': 1666744623.4381087,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.state')},
'cmd_165': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751712,\n'
' "timestamp_ms" : '
'1666751712628,\n'
' "time" : "Wed Oct 26 04:35:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 779892224,\n'
' "io_kbytes" : 761613,\n'
' "bw_bytes" : 111397260,\n'
' "bw" : 108786,\n'
' "iops" : 217572.489644,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1523225,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1392,\n'
' "max" : 26299,\n'
' "mean" : 1533.941948,\n'
' "stddev" : '
'145.787725,\n'
' "N" : 1523226\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 401,\n'
' "max" : 140834,\n'
' "mean" : 7415.475688,\n'
' "stddev" : '
'1799.292095,\n'
' "N" : 1523226,\n'
' "percentile" : {\n'
' "1.000000" : 6880,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 6944,\n'
' "20.000000" : 7008,\n'
' "30.000000" : 7072,\n'
' "40.000000" : 7072,\n'
' "50.000000" : 7136,\n'
' "60.000000" : 7200,\n'
' "70.000000" : 7264,\n'
' "80.000000" : 7328,\n'
' "90.000000" : 7392,\n'
' "95.000000" : 7648,\n'
' "99.000000" : '
'15808,\n'
' "99.500000" : '
'19328,\n'
' "99.900000" : '
'21120,\n'
' "99.950000" : '
'25472,\n'
' "99.990000" : 64768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5911,\n'
' "max" : 142467,\n'
' "mean" : 8949.417635,\n'
' "stddev" : '
'1803.451576,\n'
' "N" : 1523226\n'
' },\n'
' "bw_min" : 108527,\n'
' "bw_max" : 109183,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'108831.785714,\n'
' "bw_dev" : 172.703471,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 217054,\n'
' "iops_max" : 218366,\n'
' "iops_mean" : '
'217663.714286,\n'
' "iops_stddev" : '
'345.357881,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 23.371429,\n'
' "sys_cpu" : 54.428571,\n'
' "ctx" : 761236,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 97.033400,\n'
' "20" : 2.567119,\n'
' "50" : 0.367510,\n'
' "100" : 0.019761,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2179709,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10715,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10715,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=141045: Wed '
'Oct 26 04:35:12 2022\n'
' read: IOPS=218k, BW=106MiB/s '
'(111MB/s)(744MiB/7001msec)\n'
' slat (nsec): min=1392, '
'max=26299, avg=1533.94, '
'stdev=145.79\n'
' clat (nsec): min=401, '
'max=140834, avg=7415.48, '
'stdev=1799.29\n'
' lat (usec): min=5, max=142, '
'avg= 8.95, stdev= 1.80\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6880], '
'5.00th=[ 6944], 10.00th=[ 6944], '
'20.00th=[ 7008],\n'
' | 30.00th=[ 7072], '
'40.00th=[ 7072], 50.00th=[ '
'7136], 60.00th=[ 7200],\n'
' | 70.00th=[ 7264], '
'80.00th=[ 7328], 90.00th=[ '
'7392], 95.00th=[ 7648],\n'
' | 99.00th=[15808], '
'99.50th=[19328], '
'99.90th=[21120], '
'99.95th=[25472],\n'
' | 99.99th=[64768]\n'
' bw ( KiB/s): min=108527, '
'max=109183, per=100.00%, '
'avg=108831.79, stdev=172.70, '
'samples=14\n'
' iops : min=217054, '
'max=218366, avg=217663.71, '
'stdev=345.36, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=97.03%, 20=2.57%, 50=0.37%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=23.37%, '
'sys=54.43%, ctx=761236, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1523225,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=106MiB/s (111MB/s), '
'106MiB/s-106MiB/s '
'(111MB/s-111MB/s), io=744MiB '
'(780MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2179709/0, '
'merge=0/0, ticks=10715/0, '
'in_queue=10715, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.output'),
'state': {'begin': 1666744623.4409828,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.785879373550415,
'end': 1666744624.2268622,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.state')},
'cmd_166': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.output'),
'state': {'begin': 1666744625.21472,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7819104194641113,
'end': 1666744625.9966304,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.state')},
'cmd_167': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.output'),
'state': {'begin': 1666744625.9987113,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.078730583190918,
'end': 1666744637.077442,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.state')},
'cmd_168': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751726,\n'
' "timestamp_ms" : '
'1666751726267,\n'
' "time" : "Wed Oct 26 04:35:26 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 770420224,\n'
' "io_kbytes" : 752363,\n'
' "bw_bytes" : 110044311,\n'
' "bw" : 107465,\n'
' "iops" : 214930.009999,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1504725,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1412,\n'
' "max" : 15820,\n'
' "mean" : 1557.443289,\n'
' "stddev" : '
'138.168896,\n'
' "N" : 1504726\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 400,\n'
' "max" : 123452,\n'
' "mean" : 7487.520184,\n'
' "stddev" : '
'1736.521261,\n'
' "N" : 1504726,\n'
' "percentile" : {\n'
' "1.000000" : 6944,\n'
' "5.000000" : 7008,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7136,\n'
' "40.000000" : 7200,\n'
' "50.000000" : 7264,\n'
' "60.000000" : 7264,\n'
' "70.000000" : 7328,\n'
' "80.000000" : 7392,\n'
' "90.000000" : 7456,\n'
' "95.000000" : 7712,\n'
' "99.000000" : '
'15680,\n'
' "99.500000" : '
'19328,\n'
' "99.900000" : '
'20608,\n'
' "99.950000" : '
'24448,\n'
' "99.990000" : 64256\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5961,\n'
' "max" : 125115,\n'
' "mean" : 9044.963474,\n'
' "stddev" : '
'1739.023859,\n'
' "N" : 1504726\n'
' },\n'
' "bw_min" : 107304,\n'
' "bw_max" : 107838,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'107519.384615,\n'
' "bw_dev" : 142.703970,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 214608,\n'
' "iops_max" : 215674,\n'
' "iops_mean" : '
'215038.615385,\n'
' "iops_stddev" : '
'285.036120,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 23.071429,\n'
' "sys_cpu" : 55.214286,\n'
' "ctx" : 751974,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.981841,\n'
' "20" : 2.622007,\n'
' "50" : 0.365117,\n'
' "100" : 0.019671,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2102056,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10370,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10370,\n'
' "util" : 99.085923\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=141084: Wed '
'Oct 26 04:35:26 2022\n'
' read: IOPS=215k, BW=105MiB/s '
'(110MB/s)(735MiB/7001msec)\n'
' slat (nsec): min=1412, '
'max=15820, avg=1557.44, '
'stdev=138.17\n'
' clat (nsec): min=400, '
'max=123452, avg=7487.52, '
'stdev=1736.52\n'
' lat (usec): min=5, max=125, '
'avg= 9.04, stdev= 1.74\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6944], '
'5.00th=[ 7008], 10.00th=[ 7008], '
'20.00th=[ 7072],\n'
' | 30.00th=[ 7136], '
'40.00th=[ 7200], 50.00th=[ '
'7264], 60.00th=[ 7264],\n'
' | 70.00th=[ 7328], '
'80.00th=[ 7392], 90.00th=[ '
'7456], 95.00th=[ 7712],\n'
' | 99.00th=[15680], '
'99.50th=[19328], '
'99.90th=[20608], '
'99.95th=[24448],\n'
' | 99.99th=[64256]\n'
' bw ( KiB/s): min=107304, '
'max=107838, per=100.00%, '
'avg=107519.38, stdev=142.70, '
'samples=13\n'
' iops : min=214608, '
'max=215674, avg=215038.62, '
'stdev=285.04, samples=13\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.98%, 20=2.62%, 50=0.37%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=23.07%, '
'sys=55.21%, ctx=751974, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1504725,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=105MiB/s (110MB/s), '
'105MiB/s-105MiB/s '
'(110MB/s-110MB/s), io=735MiB '
'(770MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2102056/0, '
'merge=0/0, ticks=10370/0, '
'in_queue=10370, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.output'),
'state': {'begin': 1666744637.0800567,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7916226387023926,
'end': 1666744637.8716793,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.state')},
'cmd_169': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.output'),
'state': {'begin': 1666744638.8652065,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7901461124420166,
'end': 1666744639.6553526,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.state')},
'cmd_17': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.output'),
'state': {'begin': 1666743908.0688016,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.053016185760498,
'end': 1666743919.1218178,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.state')},
'cmd_170': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.output'),
'state': {'begin': 1666744639.65762,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.073652029037476,
'end': 1666744650.731272,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.state')},
'cmd_171': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751739,\n'
' "timestamp_ms" : '
'1666751739922,\n'
' "time" : "Wed Oct 26 04:35:39 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 771123712,\n'
' "io_kbytes" : 753050,\n'
' "bw_bytes" : 110144795,\n'
' "bw" : 107563,\n'
' "iops" : 215126.410513,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1506100,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1392,\n'
' "max" : 26369,\n'
' "mean" : 1556.186424,\n'
' "stddev" : '
'150.779054,\n'
' "N" : 1506100\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 420,\n'
' "max" : 149550,\n'
' "mean" : 7484.908665,\n'
' "stddev" : '
'1738.332903,\n'
' "N" : 1506101,\n'
' "percentile" : {\n'
' "1.000000" : 6944,\n'
' "5.000000" : 6944,\n'
' "10.000000" : 7008,\n'
' "20.000000" : 7072,\n'
' "30.000000" : 7136,\n'
' "40.000000" : 7200,\n'
' "50.000000" : 7264,\n'
' "60.000000" : 7264,\n'
' "70.000000" : 7328,\n'
' "80.000000" : 7392,\n'
' "90.000000" : 7520,\n'
' "95.000000" : 7776,\n'
' "99.000000" : '
'15680,\n'
' "99.500000" : '
'18816,\n'
' "99.900000" : '
'20864,\n'
' "99.950000" : '
'24960,\n'
' "99.990000" : 62208\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5911,\n'
' "max" : 151223,\n'
' "mean" : 9041.095007,\n'
' "stddev" : '
'1740.860134,\n'
' "N" : 1506101\n'
' },\n'
' "bw_min" : 107410,\n'
' "bw_max" : 107798,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'107620.692308,\n'
' "bw_dev" : 114.054362,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 214820,\n'
' "iops_max" : 215596,\n'
' "iops_mean" : '
'215241.230769,\n'
' "iops_stddev" : '
'227.980319,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 25.185714,\n'
' "sys_cpu" : 52.814286,\n'
' "ctx" : 752769,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.987119,\n'
' "20" : 2.608924,\n'
' "50" : 0.373880,\n'
' "100" : 0.017197,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2105479,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 10382,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 10382,\n'
' "util" : 99.085923\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=141122: Wed '
'Oct 26 04:35:39 2022\n'
' read: IOPS=215k, BW=105MiB/s '
'(110MB/s)(735MiB/7001msec)\n'
' slat (nsec): min=1392, '
'max=26369, avg=1556.19, '
'stdev=150.78\n'
' clat (nsec): min=420, '
'max=149550, avg=7484.91, '
'stdev=1738.33\n'
' lat (usec): min=5, max=151, '
'avg= 9.04, stdev= 1.74\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6944], '
'5.00th=[ 6944], 10.00th=[ 7008], '
'20.00th=[ 7072],\n'
' | 30.00th=[ 7136], '
'40.00th=[ 7200], 50.00th=[ '
'7264], 60.00th=[ 7264],\n'
' | 70.00th=[ 7328], '
'80.00th=[ 7392], 90.00th=[ '
'7520], 95.00th=[ 7776],\n'
' | 99.00th=[15680], '
'99.50th=[18816], '
'99.90th=[20864], '
'99.95th=[24960],\n'
' | 99.99th=[62208]\n'
' bw ( KiB/s): min=107410, '
'max=107798, per=100.00%, '
'avg=107620.69, stdev=114.05, '
'samples=13\n'
' iops : min=214820, '
'max=215596, avg=215241.23, '
'stdev=227.98, samples=13\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.99%, 20=2.61%, 50=0.37%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=25.19%, '
'sys=52.81%, ctx=752769, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1506100,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=105MiB/s (110MB/s), '
'105MiB/s-105MiB/s '
'(110MB/s-110MB/s), io=735MiB '
'(771MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2105479/0, '
'merge=0/0, ticks=10382/0, '
'in_queue=10382, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.output'),
'state': {'begin': 1666744650.734316,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7895231246948242,
'end': 1666744651.5238392,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.state')},
'cmd_172': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.output'),
'state': {'begin': 1666744653.5630257,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.792830228805542,
'end': 1666744654.355856,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.state')},
'cmd_173': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.output'),
'state': {'begin': 1666744654.3583882,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_aio_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.524359941482544,
'end': 1666744665.8827481,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.state')},
'cmd_174': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751754,\n'
' "timestamp_ms" : '
'1666751754974,\n'
' "time" : "Wed Oct 26 04:35:54 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 925439488,\n'
' "io_kbytes" : 903749,\n'
' "bw_bytes" : 132186757,\n'
' "bw" : 129088,\n'
' "iops" : 258177.117555,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1807498,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1603,\n'
' "max" : 30868,\n'
' "mean" : 2315.003258,\n'
' "stddev" : '
'734.458639,\n'
' "N" : 1807498\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 220,\n'
' "max" : 153418,\n'
' "mean" : 5163.300420,\n'
' "stddev" : '
'1884.570066,\n'
' "N" : 1807499,\n'
' "percentile" : {\n'
' "1.000000" : 4128,\n'
' "5.000000" : 4128,\n'
' "10.000000" : 4192,\n'
' "20.000000" : 4192,\n'
' "30.000000" : 4256,\n'
' "40.000000" : 4384,\n'
' "50.000000" : 5344,\n'
' "60.000000" : 5408,\n'
' "70.000000" : 5536,\n'
' "80.000000" : 5600,\n'
' "90.000000" : 5792,\n'
' "95.000000" : 5920,\n'
' "99.000000" : '
'13376,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'20608,\n'
' "99.950000" : '
'20864,\n'
' "99.990000" : 63744\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5731,\n'
' "max" : 155091,\n'
' "mean" : 7478.303306,\n'
' "stddev" : '
'1803.617973,\n'
' "N" : 1807499\n'
' },\n'
' "bw_min" : 128902,\n'
' "bw_max" : 129385,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'129123.500000,\n'
' "bw_dev" : 156.196006,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 257804,\n'
' "iops_max" : 258770,\n'
' "iops_mean" : '
'258247.000000,\n'
' "iops_stddev" : '
'312.392012,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 57.414286,\n'
' "sys_cpu" : 42.557143,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 1.656212,\n'
' "2" : 98.343788,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.012172,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 97.917453,\n'
' "20" : 1.799283,\n'
' "50" : 0.244924,\n'
' "100" : 0.017317,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=141171: Wed '
'Oct 26 04:35:54 2022\n'
' read: IOPS=258k, BW=126MiB/s '
'(132MB/s)(883MiB/7001msec)\n'
' slat (nsec): min=1603, '
'max=30868, avg=2315.00, '
'stdev=734.46\n'
' clat (nsec): min=220, '
'max=153418, avg=5163.30, '
'stdev=1884.57\n'
' lat (usec): min=5, max=155, '
'avg= 7.48, stdev= 1.80\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4128], '
'5.00th=[ 4128], 10.00th=[ 4192], '
'20.00th=[ 4192],\n'
' | 30.00th=[ 4256], '
'40.00th=[ 4384], 50.00th=[ '
'5344], 60.00th=[ 5408],\n'
' | 70.00th=[ 5536], '
'80.00th=[ 5600], 90.00th=[ '
'5792], 95.00th=[ 5920],\n'
' | 99.00th=[13376], '
'99.50th=[18048], '
'99.90th=[20608], '
'99.95th=[20864],\n'
' | 99.99th=[63744]\n'
' bw ( KiB/s): min=128902, '
'max=129385, per=100.00%, '
'avg=129123.50, stdev=156.20, '
'samples=14\n'
' iops : min=257804, '
'max=258770, avg=258247.00, '
'stdev=312.39, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=97.92%, 20=1.80%, '
'50=0.24%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=57.41%, '
'sys=42.56%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=1.7%, '
'2=98.3%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1807498,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=126MiB/s (132MB/s), '
'126MiB/s-126MiB/s '
'(132MB/s-132MB/s), io=883MiB '
'(925MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.output'),
'state': {'begin': 1666744665.88565,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7863152027130127,
'end': 1666744666.6719651,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.state')},
'cmd_175': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.output'),
'state': {'begin': 1666744668.6855872,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7838649749755859,
'end': 1666744669.4694521,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.state')},
'cmd_176': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.output'),
'state': {'begin': 1666744669.4713986,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_aio_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.529305934906006,
'end': 1666744681.0007045,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.state')},
'cmd_177': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751770,\n'
' "timestamp_ms" : '
'1666751770080,\n'
' "time" : "Wed Oct 26 04:36:10 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 910187008,\n'
' "io_kbytes" : 888854,\n'
' "bw_bytes" : 130008142,\n'
' "bw" : 126961,\n'
' "iops" : 253921.868305,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1777707,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1593,\n'
' "max" : 30537,\n'
' "mean" : 2352.376895,\n'
' "stddev" : '
'772.406286,\n'
' "N" : 1777708\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 241,\n'
' "max" : 135454,\n'
' "mean" : 5251.168977,\n'
' "stddev" : '
'1853.294464,\n'
' "N" : 1777707,\n'
' "percentile" : {\n'
' "1.000000" : 4192,\n'
' "5.000000" : 4192,\n'
' "10.000000" : 4256,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4320,\n'
' "40.000000" : 4448,\n'
' "50.000000" : 5472,\n'
' "60.000000" : 5536,\n'
' "70.000000" : 5664,\n'
' "80.000000" : 5728,\n'
' "90.000000" : 5920,\n'
' "95.000000" : 5984,\n'
' "99.000000" : '
'13376,\n'
' "99.500000" : '
'17792,\n'
' "99.900000" : '
'20608,\n'
' "99.950000" : '
'20864,\n'
' "99.990000" : 60160\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5781,\n'
' "max" : 137148,\n'
' "mean" : 7603.546254,\n'
' "stddev" : '
'1761.177099,\n'
' "N" : 1777707\n'
' },\n'
' "bw_min" : 126711,\n'
' "bw_max" : 127293,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'126994.142857,\n'
' "bw_dev" : 203.450942,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 253422,\n'
' "iops_max" : 254586,\n'
' "iops_mean" : '
'253988.357143,\n'
' "iops_stddev" : '
'407.005879,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 58.271429,\n'
' "sys_cpu" : 41.700000,\n'
' "ctx" : 20,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 1.653985,\n'
' "2" : 98.346015,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.014457,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 97.943981,\n'
' "20" : 1.808003,\n'
' "50" : 0.211565,\n'
' "100" : 0.018394,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=141222: Wed '
'Oct 26 04:36:10 2022\n'
' read: IOPS=254k, BW=124MiB/s '
'(130MB/s)(868MiB/7001msec)\n'
' slat (nsec): min=1593, '
'max=30537, avg=2352.38, '
'stdev=772.41\n'
' clat (nsec): min=241, '
'max=135454, avg=5251.17, '
'stdev=1853.29\n'
' lat (usec): min=5, max=137, '
'avg= 7.60, stdev= 1.76\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4192], '
'5.00th=[ 4192], 10.00th=[ 4256], '
'20.00th=[ 4256],\n'
' | 30.00th=[ 4320], '
'40.00th=[ 4448], 50.00th=[ '
'5472], 60.00th=[ 5536],\n'
' | 70.00th=[ 5664], '
'80.00th=[ 5728], 90.00th=[ '
'5920], 95.00th=[ 5984],\n'
' | 99.00th=[13376], '
'99.50th=[17792], '
'99.90th=[20608], '
'99.95th=[20864],\n'
' | 99.99th=[60160]\n'
' bw ( KiB/s): min=126711, '
'max=127293, per=100.00%, '
'avg=126994.14, stdev=203.45, '
'samples=14\n'
' iops : min=253422, '
'max=254586, avg=253988.36, '
'stdev=407.01, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=97.94%, 20=1.81%, '
'50=0.21%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=58.27%, '
'sys=41.70%, ctx=20, majf=0, '
'minf=0\n'
' IO depths : 1=1.7%, '
'2=98.3%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1777707,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=124MiB/s (130MB/s), '
'124MiB/s-124MiB/s '
'(130MB/s-130MB/s), io=868MiB '
'(910MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.output'),
'state': {'begin': 1666744681.0033295,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7950851917266846,
'end': 1666744681.7984147,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.state')},
'cmd_178': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.output'),
'state': {'begin': 1666744683.82356,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7866497039794922,
'end': 1666744684.6102097,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.state')},
'cmd_179': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.output'),
'state': {'begin': 1666744684.6121955,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_aio_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.426987886428833,
'end': 1666744696.0391834,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.state')},
'cmd_18': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751008,\n'
' "timestamp_ms" : '
'1666751008324,\n'
' "time" : "Wed Oct 26 04:23:28 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 291316224,\n'
' "io_kbytes" : 284488,\n'
' "bw_bytes" : 41610659,\n'
' "bw" : 40635,\n'
' "iops" : 81270.675618,\n'
' "runtime" : 7001,\n'
' "total_ios" : 568976,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2325,\n'
' "max" : 28083,\n'
' "mean" : 2529.571946,\n'
' "stddev" : 230.071012,\n'
' "N" : 568977\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 270,\n'
' "max" : 111159,\n'
' "mean" : 9106.443340,\n'
' "stddev" : '
'1251.821492,\n'
' "N" : 568976,\n'
' "percentile" : {\n'
' "1.000000" : 8768,\n'
' "5.000000" : 8768,\n'
' "10.000000" : 8768,\n'
' "20.000000" : 8896,\n'
' "30.000000" : 8896,\n'
' "40.000000" : 8896,\n'
' "50.000000" : 8896,\n'
' "60.000000" : 8896,\n'
' "70.000000" : 9024,\n'
' "80.000000" : 9024,\n'
' "90.000000" : 9024,\n'
' "95.000000" : 9280,\n'
' "99.000000" : 15552,\n'
' "99.500000" : 16512,\n'
' "99.900000" : 19072,\n'
' "99.950000" : 21632,\n'
' "99.990000" : 55552\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8857,\n'
' "max" : 113743,\n'
' "mean" : 11636.015347,\n'
' "stddev" : '
'1270.086719,\n'
' "N" : 568976\n'
' },\n'
' "bw_min" : 40559,\n'
' "bw_max" : 40758,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'40646.000000,\n'
' "bw_dev" : 70.886692,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 81118,\n'
' "iops_max" : 81516,\n'
' "iops_mean" : '
'81292.000000,\n'
' "iops_stddev" : '
'141.773384,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.200000,\n'
' "sys_cpu" : 43.085714,\n'
' "ctx" : 568842,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.012479,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.371067,\n'
' "20" : 2.537365,\n'
' "50" : 0.064678,\n'
' "100" : 0.014236,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=138867: Wed '
'Oct 26 04:23:28 2022\n'
' read: IOPS=81.3k, BW=39.7MiB/s '
'(41.6MB/s)(278MiB/7001msec)\n'
' slat (nsec): min=2325, '
'max=28083, avg=2529.57, '
'stdev=230.07\n'
' clat (nsec): min=270, '
'max=111159, avg=9106.44, '
'stdev=1251.82\n'
' lat (usec): min=8, max=113, '
'avg=11.64, stdev= 1.27\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8768], 5.00th=[ '
'8768], 10.00th=[ 8768], 20.00th=[ '
'8896],\n'
' | 30.00th=[ 8896], 40.00th=[ '
'8896], 50.00th=[ 8896], 60.00th=[ '
'8896],\n'
' | 70.00th=[ 9024], 80.00th=[ '
'9024], 90.00th=[ 9024], 95.00th=[ '
'9280],\n'
' | 99.00th=[15552], '
'99.50th=[16512], 99.90th=[19072], '
'99.95th=[21632],\n'
' | 99.99th=[55552]\n'
' bw ( KiB/s): min=40559, '
'max=40758, per=100.00%, '
'avg=40646.00, stdev=70.89, '
'samples=14\n'
' iops : min=81118, '
'max=81516, avg=81292.00, '
'stdev=141.77, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=97.37%, '
'20=2.54%, 50=0.06%, 100=0.01%, '
'250=0.01%\n'
' cpu : usr=16.20%, '
'sys=43.09%, ctx=568842, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=568976,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=39.7MiB/s (41.6MB/s), '
'39.7MiB/s-39.7MiB/s '
'(41.6MB/s-41.6MB/s), io=278MiB '
'(291MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.output'),
'state': {'begin': 1666743919.1248143,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8096821308135986,
'end': 1666743919.9344964,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.state')},
'cmd_180': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751785,\n'
' "timestamp_ms" : '
'1666751785122,\n'
' "time" : "Wed Oct 26 04:36:25 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 922618368,\n'
' "io_kbytes" : 900994,\n'
' "bw_bytes" : 131783797,\n'
' "bw" : 128695,\n'
' "iops" : 257389.944294,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1801987,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1583,\n'
' "max" : 29826,\n'
' "mean" : 2299.051626,\n'
' "stddev" : '
'743.697493,\n'
' "N" : 1801988\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 230,\n'
' "max" : 138339,\n'
' "mean" : 5203.031010,\n'
' "stddev" : '
'1872.544552,\n'
' "N" : 1801987,\n'
' "percentile" : {\n'
' "1.000000" : 4128,\n'
' "5.000000" : 4192,\n'
' "10.000000" : 4192,\n'
' "20.000000" : 4256,\n'
' "30.000000" : 4320,\n'
' "40.000000" : 4448,\n'
' "50.000000" : 5408,\n'
' "60.000000" : 5472,\n'
' "70.000000" : 5600,\n'
' "80.000000" : 5664,\n'
' "90.000000" : 5856,\n'
' "95.000000" : 5984,\n'
' "99.000000" : '
'13376,\n'
' "99.500000" : '
'18048,\n'
' "99.900000" : '
'20864,\n'
' "99.950000" : '
'21120,\n'
' "99.990000" : 61184\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5740,\n'
' "max" : 140022,\n'
' "mean" : 7502.082299,\n'
' "stddev" : '
'1785.297905,\n'
' "N" : 1801987\n'
' },\n'
' "bw_min" : 128549,\n'
' "bw_max" : 129325,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'128728.785714,\n'
' "bw_dev" : 240.665161,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 257098,\n'
' "iops_max" : 258650,\n'
' "iops_mean" : '
'257457.714286,\n'
' "iops_stddev" : '
'481.220792,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 59.585714,\n'
' "sys_cpu" : 40.385714,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 1.640911,\n'
' "2" : 98.359089,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.011876,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 97.912249,\n'
' "20" : 1.800734,\n'
' "50" : 0.250335,\n'
' "100" : 0.017148,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=141271: Wed '
'Oct 26 04:36:25 2022\n'
' read: IOPS=257k, BW=126MiB/s '
'(132MB/s)(880MiB/7001msec)\n'
' slat (nsec): min=1583, '
'max=29826, avg=2299.05, '
'stdev=743.70\n'
' clat (nsec): min=230, '
'max=138339, avg=5203.03, '
'stdev=1872.54\n'
' lat (usec): min=5, max=140, '
'avg= 7.50, stdev= 1.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4128], '
'5.00th=[ 4192], 10.00th=[ 4192], '
'20.00th=[ 4256],\n'
' | 30.00th=[ 4320], '
'40.00th=[ 4448], 50.00th=[ '
'5408], 60.00th=[ 5472],\n'
' | 70.00th=[ 5600], '
'80.00th=[ 5664], 90.00th=[ '
'5856], 95.00th=[ 5984],\n'
' | 99.00th=[13376], '
'99.50th=[18048], '
'99.90th=[20864], '
'99.95th=[21120],\n'
' | 99.99th=[61184]\n'
' bw ( KiB/s): min=128549, '
'max=129325, per=100.00%, '
'avg=128728.79, stdev=240.67, '
'samples=14\n'
' iops : min=257098, '
'max=258650, avg=257457.71, '
'stdev=481.22, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=97.91%, 20=1.80%, '
'50=0.25%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=59.59%, '
'sys=40.39%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=1.6%, '
'2=98.4%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1801987,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=126MiB/s (132MB/s), '
'126MiB/s-126MiB/s '
'(132MB/s-132MB/s), io=880MiB '
'(923MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.output'),
'state': {'begin': 1666744696.0420241,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7890415191650391,
'end': 1666744696.8310657,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.state')},
'cmd_181': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.output'),
'state': {'begin': 1666744698.8490248,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7922971248626709,
'end': 1666744699.641322,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.state')},
'cmd_182': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.output'),
'state': {'begin': 1666744699.6432898,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.539600133895874,
'end': 1666744711.18289,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.state')},
'cmd_183': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751800,\n'
' "timestamp_ms" : '
'1666751800261,\n'
' "time" : "Wed Oct 26 04:36:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 892137472,\n'
' "io_kbytes" : 871228,\n'
' "bw_bytes" : 127430005,\n'
' "bw" : 124443,\n'
' "iops" : 248886.444794,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1742454,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1593,\n'
' "max" : 19317,\n'
' "mean" : 2306.401335,\n'
' "stddev" : '
'734.829566,\n'
' "N" : 1742455\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 591,\n'
' "max" : 121829,\n'
' "mean" : 5472.362839,\n'
' "stddev" : '
'1813.157116,\n'
' "N" : 1742454,\n'
' "percentile" : {\n'
' "1.000000" : 4448,\n'
' "5.000000" : 4512,\n'
' "10.000000" : 4512,\n'
' "20.000000" : 4576,\n'
' "30.000000" : 4576,\n'
' "40.000000" : 4640,\n'
' "50.000000" : 5728,\n'
' "60.000000" : 5792,\n'
' "70.000000" : 5856,\n'
' "80.000000" : 5920,\n'
' "90.000000" : 5984,\n'
' "95.000000" : 6176,\n'
' "99.000000" : '
'13248,\n'
' "99.500000" : '
'17536,\n'
' "99.900000" : '
'20352,\n'
' "99.950000" : '
'20864,\n'
' "99.990000" : 62208\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5901,\n'
' "max" : 123492,\n'
' "mean" : 7778.763796,\n'
' "stddev" : '
'1710.924115,\n'
' "N" : 1742454\n'
' },\n'
' "bw_min" : 124145,\n'
' "bw_max" : 124816,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'124484.384615,\n'
' "bw_dev" : 207.810626,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 248290,\n'
' "iops_max" : 249632,\n'
' "iops_mean" : '
'248968.692308,\n'
' "iops_stddev" : '
'415.787483,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.300000,\n'
' "sys_cpu" : 60.671429,\n'
' "ctx" : 18,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 1.483425,\n'
' "2" : 98.516575,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.013200,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.015098,\n'
' "20" : 1.779043,\n'
' "50" : 0.169072,\n'
' "100" : 0.019513,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141320: Wed '
'Oct 26 04:36:40 2022\n'
' read: IOPS=249k, BW=122MiB/s '
'(127MB/s)(851MiB/7001msec)\n'
' slat (nsec): min=1593, '
'max=19317, avg=2306.40, '
'stdev=734.83\n'
' clat (nsec): min=591, '
'max=121829, avg=5472.36, '
'stdev=1813.16\n'
' lat (usec): min=5, max=123, '
'avg= 7.78, stdev= 1.71\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4448], '
'5.00th=[ 4512], 10.00th=[ 4512], '
'20.00th=[ 4576],\n'
' | 30.00th=[ 4576], '
'40.00th=[ 4640], 50.00th=[ '
'5728], 60.00th=[ 5792],\n'
' | 70.00th=[ 5856], '
'80.00th=[ 5920], 90.00th=[ '
'5984], 95.00th=[ 6176],\n'
' | 99.00th=[13248], '
'99.50th=[17536], '
'99.90th=[20352], '
'99.95th=[20864],\n'
' | 99.99th=[62208]\n'
' bw ( KiB/s): min=124145, '
'max=124816, per=100.00%, '
'avg=124484.38, stdev=207.81, '
'samples=13\n'
' iops : min=248290, '
'max=249632, avg=248968.69, '
'stdev=415.79, samples=13\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.02%, 20=1.78%, '
'50=0.17%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=39.30%, '
'sys=60.67%, ctx=18, majf=0, '
'minf=0\n'
' IO depths : 1=1.5%, '
'2=98.5%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1742454,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=122MiB/s (127MB/s), '
'122MiB/s-122MiB/s '
'(127MB/s-127MB/s), io=851MiB '
'(892MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.output'),
'state': {'begin': 1666744711.1857076,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7863950729370117,
'end': 1666744711.9721026,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.state')},
'cmd_184': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.output'),
'state': {'begin': 1666744713.9869568,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7969717979431152,
'end': 1666744714.7839286,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.state')},
'cmd_185': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.output'),
'state': {'begin': 1666744714.7859266,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.573662757873535,
'end': 1666744726.3595893,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.state')},
'cmd_186': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751815,\n'
' "timestamp_ms" : '
'1666751815448,\n'
' "time" : "Wed Oct 26 04:36:55 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 888960000,\n'
' "io_kbytes" : 868125,\n'
' "bw_bytes" : 126976146,\n'
' "bw" : 124000,\n'
' "iops" : 248000.142837,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1736249,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1613,\n'
' "max" : 33963,\n'
' "mean" : 2325.591072,\n'
' "stddev" : '
'759.135311,\n'
' "N" : 1736249\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 621,\n'
' "max" : 121107,\n'
' "mean" : 5472.580111,\n'
' "stddev" : '
'1812.845915,\n'
' "N" : 1736250,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 4320,\n'
' "10.000000" : 4512,\n'
' "20.000000" : 4576,\n'
' "30.000000" : 4640,\n'
' "40.000000" : 4704,\n'
' "50.000000" : 5472,\n'
' "60.000000" : 5856,\n'
' "70.000000" : 5920,\n'
' "80.000000" : 5984,\n'
' "90.000000" : 6048,\n'
' "95.000000" : 6240,\n'
' "99.000000" : '
'13376,\n'
' "99.500000" : '
'17280,\n'
' "99.900000" : '
'20096,\n'
' "99.950000" : '
'20608,\n'
' "99.990000" : 60672\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5891,\n'
' "max" : 122811,\n'
' "mean" : 7798.170796,\n'
' "stddev" : '
'1715.249598,\n'
' "N" : 1736250\n'
' },\n'
' "bw_min" : 123725,\n'
' "bw_max" : 124153,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'124029.571429,\n'
' "bw_dev" : 106.987213,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 247450,\n'
' "iops_max" : 248306,\n'
' "iops_mean" : '
'248059.000000,\n'
' "iops_stddev" : '
'214.096671,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 35.585714,\n'
' "sys_cpu" : 64.385714,\n'
' "ctx" : 24,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 1.473003,\n'
' "2" : 98.526997,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.012268,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 97.911532,\n'
' "20" : 1.955998,\n'
' "50" : 0.096991,\n'
' "100" : 0.019410,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141369: Wed '
'Oct 26 04:36:55 2022\n'
' read: IOPS=248k, BW=121MiB/s '
'(127MB/s)(848MiB/7001msec)\n'
' slat (nsec): min=1613, '
'max=33963, avg=2325.59, '
'stdev=759.14\n'
' clat (nsec): min=621, '
'max=121107, avg=5472.58, '
'stdev=1812.85\n'
' lat (usec): min=5, max=122, '
'avg= 7.80, stdev= 1.72\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], '
'5.00th=[ 4320], 10.00th=[ 4512], '
'20.00th=[ 4576],\n'
' | 30.00th=[ 4640], '
'40.00th=[ 4704], 50.00th=[ '
'5472], 60.00th=[ 5856],\n'
' | 70.00th=[ 5920], '
'80.00th=[ 5984], 90.00th=[ '
'6048], 95.00th=[ 6240],\n'
' | 99.00th=[13376], '
'99.50th=[17280], '
'99.90th=[20096], '
'99.95th=[20608],\n'
' | 99.99th=[60672]\n'
' bw ( KiB/s): min=123725, '
'max=124153, per=100.00%, '
'avg=124029.57, stdev=106.99, '
'samples=14\n'
' iops : min=247450, '
'max=248306, avg=248059.00, '
'stdev=214.10, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=97.91%, 20=1.96%, '
'50=0.10%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=35.59%, '
'sys=64.39%, ctx=24, majf=0, '
'minf=0\n'
' IO depths : 1=1.5%, '
'2=98.5%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1736249,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=121MiB/s (127MB/s), '
'121MiB/s-121MiB/s '
'(127MB/s-127MB/s), io=848MiB '
'(889MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.output'),
'state': {'begin': 1666744726.362394,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.793710470199585,
'end': 1666744727.1561046,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.state')},
'cmd_187': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.output'),
'state': {'begin': 1666744729.1706247,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7985272407531738,
'end': 1666744729.969152,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.state')},
'cmd_188': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.output'),
'state': {'begin': 1666744729.971215,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.529080152511597,
'end': 1666744741.5002952,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.state')},
'cmd_189': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751830,\n'
' "timestamp_ms" : '
'1666751830581,\n'
' "time" : "Wed Oct 26 04:37:10 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 886879232,\n'
' "io_kbytes" : 866093,\n'
' "bw_bytes" : 126678936,\n'
' "bw" : 123709,\n'
' "iops" : 247419.511498,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1732184,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1623,\n'
' "max" : 26189,\n'
' "mean" : 2344.885200,\n'
' "stddev" : '
'746.431565,\n'
' "N" : 1732185\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 621,\n'
' "max" : 175389,\n'
' "mean" : 5470.346042,\n'
' "stddev" : '
'1841.692980,\n'
' "N" : 1732184,\n'
' "percentile" : {\n'
' "1.000000" : 4256,\n'
' "5.000000" : 4320,\n'
' "10.000000" : 4512,\n'
' "20.000000" : 4576,\n'
' "30.000000" : 4640,\n'
' "40.000000" : 4704,\n'
' "50.000000" : 5536,\n'
' "60.000000" : 5856,\n'
' "70.000000" : 5920,\n'
' "80.000000" : 5984,\n'
' "90.000000" : 6048,\n'
' "95.000000" : 6240,\n'
' "99.000000" : '
'13120,\n'
' "99.500000" : '
'17024,\n'
' "99.900000" : '
'20096,\n'
' "99.950000" : '
'20352,\n'
' "99.990000" : 64768\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5921,\n'
' "max" : 177122,\n'
' "mean" : 7815.230866,\n'
' "stddev" : '
'1740.065131,\n'
' "N" : 1732184\n'
' },\n'
' "bw_min" : 123454,\n'
' "bw_max" : 124192,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'123727.230769,\n'
' "bw_dev" : 248.715753,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 246908,\n'
' "iops_max" : 248384,\n'
' "iops_mean" : '
'247454.461538,\n'
' "iops_stddev" : '
'497.431505,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 37.471429,\n'
' "sys_cpu" : 62.500000,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 1.468666,\n'
' "2" : 98.531334,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010045,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.048995,\n'
' "20" : 1.830464,\n'
' "50" : 0.086019,\n'
' "100" : 0.019109,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141419: Wed '
'Oct 26 04:37:10 2022\n'
' read: IOPS=247k, BW=121MiB/s '
'(127MB/s)(846MiB/7001msec)\n'
' slat (nsec): min=1623, '
'max=26189, avg=2344.89, '
'stdev=746.43\n'
' clat (nsec): min=621, '
'max=175389, avg=5470.35, '
'stdev=1841.69\n'
' lat (usec): min=5, max=177, '
'avg= 7.82, stdev= 1.74\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4256], '
'5.00th=[ 4320], 10.00th=[ 4512], '
'20.00th=[ 4576],\n'
' | 30.00th=[ 4640], '
'40.00th=[ 4704], 50.00th=[ '
'5536], 60.00th=[ 5856],\n'
' | 70.00th=[ 5920], '
'80.00th=[ 5984], 90.00th=[ '
'6048], 95.00th=[ 6240],\n'
' | 99.00th=[13120], '
'99.50th=[17024], '
'99.90th=[20096], '
'99.95th=[20352],\n'
' | 99.99th=[64768]\n'
' bw ( KiB/s): min=123454, '
'max=124192, per=100.00%, '
'avg=123727.23, stdev=248.72, '
'samples=13\n'
' iops : min=246908, '
'max=248384, avg=247454.46, '
'stdev=497.43, samples=13\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.05%, 20=1.83%, '
'50=0.09%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=37.47%, '
'sys=62.50%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=1.5%, '
'2=98.5%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1732184,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=121MiB/s (127MB/s), '
'121MiB/s-121MiB/s '
'(127MB/s-127MB/s), io=846MiB '
'(887MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.output'),
'state': {'begin': 1666744741.5030072,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7861819267272949,
'end': 1666744742.289189,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.state')},
'cmd_19': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.output'),
'state': {'begin': 1666743921.9537325,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7813303470611572,
'end': 1666743922.7350628,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.state')},
'cmd_190': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.output'),
'state': {'begin': 1666744743.2813637,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7844269275665283,
'end': 1666744744.0657907,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.state')},
'cmd_191': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.output'),
'state': {'begin': 1666744744.0681014,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.10428261756897,
'end': 1666744755.172384,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.state')},
'cmd_192': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751844,\n'
' "timestamp_ms" : '
'1666751844361,\n'
' "time" : "Wed Oct 26 04:37:24 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 727684608,\n'
' "io_kbytes" : 710629,\n'
' "bw_bytes" : 103940095,\n'
' "bw" : 101503,\n'
' "iops" : 203007.713184,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1421257,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1472,\n'
' "max" : 38092,\n'
' "mean" : 1673.767642,\n'
' "stddev" : '
'256.286057,\n'
' "N" : 1421258\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 471,\n'
' "max" : 160200,\n'
' "mean" : 7924.788124,\n'
' "stddev" : '
'1919.052306,\n'
' "N" : 1421258,\n'
' "percentile" : {\n'
' "1.000000" : 4704,\n'
' "5.000000" : 7328,\n'
' "10.000000" : 7520,\n'
' "20.000000" : 7584,\n'
' "30.000000" : 7648,\n'
' "40.000000" : 7712,\n'
' "50.000000" : 7776,\n'
' "60.000000" : 7840,\n'
' "70.000000" : 7904,\n'
' "80.000000" : 7968,\n'
' "90.000000" : 8032,\n'
' "95.000000" : 8160,\n'
' "99.000000" : '
'15552,\n'
' "99.500000" : '
'19584,\n'
' "99.900000" : '
'25216,\n'
' "99.950000" : '
'25472,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5992,\n'
' "max" : 162054,\n'
' "mean" : 9598.555765,\n'
' "stddev" : '
'1912.374899,\n'
' "N" : 1421258\n'
' },\n'
' "bw_min" : 101229,\n'
' "bw_max" : 101973,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'101532.571429,\n'
' "bw_dev" : 208.463136,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 202458,\n'
' "iops_max" : 203946,\n'
' "iops_mean" : '
'203065.214286,\n'
' "iops_stddev" : '
'416.847542,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 22.600000,\n'
' "sys_cpu" : 57.742857,\n'
' "ctx" : 681640,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.858907,\n'
' "20" : 2.760233,\n'
' "50" : 0.346454,\n'
' "100" : 0.023078,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141458: Wed '
'Oct 26 04:37:24 2022\n'
' read: IOPS=203k, BW=99.1MiB/s '
'(104MB/s)(694MiB/7001msec)\n'
' slat (nsec): min=1472, '
'max=38092, avg=1673.77, '
'stdev=256.29\n'
' clat (nsec): min=471, '
'max=160200, avg=7924.79, '
'stdev=1919.05\n'
' lat (usec): min=5, max=162, '
'avg= 9.60, stdev= 1.91\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4704], '
'5.00th=[ 7328], 10.00th=[ 7520], '
'20.00th=[ 7584],\n'
' | 30.00th=[ 7648], '
'40.00th=[ 7712], 50.00th=[ '
'7776], 60.00th=[ 7840],\n'
' | 70.00th=[ 7904], '
'80.00th=[ 7968], 90.00th=[ '
'8032], 95.00th=[ 8160],\n'
' | 99.00th=[15552], '
'99.50th=[19584], '
'99.90th=[25216], '
'99.95th=[25472],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=101229, '
'max=101973, per=100.00%, '
'avg=101532.57, stdev=208.46, '
'samples=14\n'
' iops : min=202458, '
'max=203946, avg=203065.21, '
'stdev=416.85, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.86%, 20=2.76%, 50=0.35%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=22.60%, '
'sys=57.74%, ctx=681640, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1421257,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=99.1MiB/s (104MB/s), '
'99.1MiB/s-99.1MiB/s '
'(104MB/s-104MB/s), io=694MiB '
'(728MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.output'),
'state': {'begin': 1666744755.1754644,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7867915630340576,
'end': 1666744755.962256,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.state')},
'cmd_193': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.output'),
'state': {'begin': 1666744756.9621143,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7869017124176025,
'end': 1666744757.749016,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.state')},
'cmd_194': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.output'),
'state': {'begin': 1666744757.7510684,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.106256008148193,
'end': 1666744768.8573244,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.state')},
'cmd_195': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751858,\n'
' "timestamp_ms" : '
'1666751858044,\n'
' "time" : "Wed Oct 26 04:37:38 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 724207104,\n'
' "io_kbytes" : 707233,\n'
' "bw_bytes" : 103443380,\n'
' "bw" : 101018,\n'
' "iops" : 202037.566062,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1414465,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1472,\n'
' "max" : 29145,\n'
' "mean" : 1659.259898,\n'
' "stddev" : '
'223.201329,\n'
' "N" : 1414466\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 451,\n'
' "max" : 122610,\n'
' "mean" : 7996.446592,\n'
' "stddev" : '
'1863.055498,\n'
' "N" : 1414466,\n'
' "percentile" : {\n'
' "1.000000" : 4896,\n'
' "5.000000" : 7520,\n'
' "10.000000" : 7584,\n'
' "20.000000" : 7584,\n'
' "30.000000" : 7648,\n'
' "40.000000" : 7712,\n'
' "50.000000" : 7840,\n'
' "60.000000" : 7840,\n'
' "70.000000" : 7904,\n'
' "80.000000" : 7968,\n'
' "90.000000" : 8032,\n'
' "95.000000" : 8256,\n'
' "99.000000" : '
'15680,\n'
' "99.500000" : '
'19584,\n'
' "99.900000" : '
'25472,\n'
' "99.950000" : '
'26240,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5992,\n'
' "max" : 124454,\n'
' "mean" : 9655.706490,\n'
' "stddev" : '
'1864.008513,\n'
' "N" : 1414466\n'
' },\n'
' "bw_min" : 100860,\n'
' "bw_max" : 101305,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'101042.214286,\n'
' "bw_dev" : 134.849305,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 201720,\n'
' "iops_max" : 202610,\n'
' "iops_mean" : '
'202084.285714,\n'
' "iops_stddev" : '
'269.778094,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 22.514286,\n'
' "sys_cpu" : 57.471429,\n'
' "ctx" : 693903,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010888,\n'
' "10" : 96.823039,\n'
' "20" : 2.759206,\n'
' "50" : 0.380002,\n'
' "100" : 0.020432,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141498: Wed '
'Oct 26 04:37:38 2022\n'
' read: IOPS=202k, BW=98.7MiB/s '
'(103MB/s)(691MiB/7001msec)\n'
' slat (nsec): min=1472, '
'max=29145, avg=1659.26, '
'stdev=223.20\n'
' clat (nsec): min=451, '
'max=122610, avg=7996.45, '
'stdev=1863.06\n'
' lat (usec): min=5, max=124, '
'avg= 9.66, stdev= 1.86\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4896], '
'5.00th=[ 7520], 10.00th=[ 7584], '
'20.00th=[ 7584],\n'
' | 30.00th=[ 7648], '
'40.00th=[ 7712], 50.00th=[ '
'7840], 60.00th=[ 7840],\n'
' | 70.00th=[ 7904], '
'80.00th=[ 7968], 90.00th=[ '
'8032], 95.00th=[ 8256],\n'
' | 99.00th=[15680], '
'99.50th=[19584], '
'99.90th=[25472], '
'99.95th=[26240],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=100860, '
'max=101305, per=100.00%, '
'avg=101042.21, stdev=134.85, '
'samples=14\n'
' iops : min=201720, '
'max=202610, avg=202084.29, '
'stdev=269.78, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.82%, 20=2.76%, 50=0.38%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=22.51%, '
'sys=57.47%, ctx=693903, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1414465,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=98.7MiB/s (103MB/s), '
'98.7MiB/s-98.7MiB/s '
'(103MB/s-103MB/s), io=691MiB '
'(724MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.output'),
'state': {'begin': 1666744768.8603215,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7948360443115234,
'end': 1666744769.6551576,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.state')},
'cmd_196': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.output'),
'state': {'begin': 1666744770.6479423,
'cmd': 'rm '
'/tmp/fio-output_512_2_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7802488803863525,
'end': 1666744771.4281912,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.state')},
'cmd_197': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.output'),
'state': {'begin': 1666744771.4306922,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="2" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_2_libaio-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.090673923492432,
'end': 1666744782.5213661,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.state')},
'cmd_198': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=2\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751871,\n'
' "timestamp_ms" : '
'1666751871710,\n'
' "time" : "Wed Oct 26 04:37:51 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "2",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 724407808,\n'
' "io_kbytes" : 707429,\n'
' "bw_bytes" : 103472047,\n'
' "bw" : 101046,\n'
' "iops" : 202093.700900,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1414858,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1462,\n'
' "max" : 24455,\n'
' "mean" : 1634.391682,\n'
' "stddev" : '
'230.781785,\n'
' "N" : 1414858\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 451,\n'
' "max" : 166613,\n'
' "mean" : 8011.332859,\n'
' "stddev" : '
'1891.589306,\n'
' "N" : 1414859,\n'
' "percentile" : {\n'
' "1.000000" : 4960,\n'
' "5.000000" : 7520,\n'
' "10.000000" : 7584,\n'
' "20.000000" : 7648,\n'
' "30.000000" : 7712,\n'
' "40.000000" : 7776,\n'
' "50.000000" : 7840,\n'
' "60.000000" : 7904,\n'
' "70.000000" : 7904,\n'
' "80.000000" : 7968,\n'
' "90.000000" : 8032,\n'
' "95.000000" : 8256,\n'
' "99.000000" : '
'15808,\n'
' "99.500000" : '
'19584,\n'
' "99.900000" : '
'25472,\n'
' "99.950000" : '
'25984,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5981,\n'
' "max" : 168406,\n'
' "mean" : 9645.724596,\n'
' "stddev" : '
'1891.318592,\n'
' "N" : 1414859\n'
' },\n'
' "bw_min" : 100815,\n'
' "bw_max" : 101425,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'101073.285714,\n'
' "bw_dev" : 170.786609,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 201630,\n'
' "iops_max" : 202851,\n'
' "iops_mean" : '
'202146.642857,\n'
' "iops_stddev" : '
'341.731700,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.114286,\n'
' "sys_cpu" : 59.771429,\n'
' "ctx" : 694144,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 100.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 96.841308,\n'
' "20" : 2.744304,\n'
' "50" : 0.377635,\n'
' "100" : 0.023041,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 2,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141538: Wed '
'Oct 26 04:37:51 2022\n'
' read: IOPS=202k, BW=98.7MiB/s '
'(103MB/s)(691MiB/7001msec)\n'
' slat (nsec): min=1462, '
'max=24455, avg=1634.39, '
'stdev=230.78\n'
' clat (nsec): min=451, '
'max=166613, avg=8011.33, '
'stdev=1891.59\n'
' lat (usec): min=5, max=168, '
'avg= 9.65, stdev= 1.89\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4960], '
'5.00th=[ 7520], 10.00th=[ 7584], '
'20.00th=[ 7648],\n'
' | 30.00th=[ 7712], '
'40.00th=[ 7776], 50.00th=[ '
'7840], 60.00th=[ 7904],\n'
' | 70.00th=[ 7904], '
'80.00th=[ 7968], 90.00th=[ '
'8032], 95.00th=[ 8256],\n'
' | 99.00th=[15808], '
'99.50th=[19584], '
'99.90th=[25472], '
'99.95th=[25984],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=100815, '
'max=101425, per=100.00%, '
'avg=101073.29, stdev=170.79, '
'samples=14\n'
' iops : min=201630, '
'max=202851, avg=202146.64, '
'stdev=341.73, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=96.84%, 20=2.74%, 50=0.38%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=20.11%, '
'sys=59.77%, ctx=694144, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, '
'2=100.0%, 4=0.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1414858,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=2\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=98.7MiB/s (103MB/s), '
'98.7MiB/s-98.7MiB/s '
'(103MB/s-103MB/s), io=691MiB '
'(724MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.output'),
'state': {'begin': 1666744782.524606,
'cmd': 'cat '
'/tmp/fio-output_512_2_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7909767627716064,
'end': 1666744783.3155828,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.state')},
'cmd_199': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.output'),
'state': {'begin': 1666744784.3118296,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.784106969833374,
'end': 1666744785.0959365,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.state')},
'cmd_20': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.output'),
'state': {'begin': 1666743922.7373698,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.47360110282898,
'end': 1666743934.2109709,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.state')},
'cmd_200': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.output'),
'state': {'begin': 1666744785.0982733,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.04575800895691,
'end': 1666744796.1440313,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.state')},
'cmd_201': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751885,\n'
' "timestamp_ms" : '
'1666751885332,\n'
' "time" : "Wed Oct 26 04:38:05 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1191164928,\n'
' "io_kbytes" : 1163247,\n'
' "bw_bytes" : 170142112,\n'
' "bw" : 166154,\n'
' "iops" : 332308.241680,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2326490,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1142,\n'
' "max" : 25858,\n'
' "mean" : 2197.265368,\n'
' "stddev" : '
'1099.926498,\n'
' "N" : 2326491\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 165000,\n'
' "mean" : 9370.234935,\n'
' "stddev" : '
'2562.392231,\n'
' "N" : 2326493,\n'
' "percentile" : {\n'
' "1.000000" : 6048,\n'
' "5.000000" : 7648,\n'
' "10.000000" : 8640,\n'
' "20.000000" : 8768,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 8768,\n'
' "50.000000" : 8896,\n'
' "60.000000" : 8896,\n'
' "70.000000" : 9024,\n'
' "80.000000" : 9152,\n'
' "90.000000" : '
'10304,\n'
' "95.000000" : '
'11584,\n'
' "99.000000" : '
'21888,\n'
' "99.500000" : '
'24192,\n'
' "99.900000" : '
'28800,\n'
' "99.950000" : '
'34048,\n'
' "99.990000" : 70144\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5481,\n'
' "max" : 166182,\n'
' "mean" : '
'11567.500667,\n'
' "stddev" : '
'2627.540346,\n'
' "N" : 2326493\n'
' },\n'
' "bw_min" : 165820,\n'
' "bw_max" : 166583,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'166195.857143,\n'
' "bw_dev" : 234.095923,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 331640,\n'
' "iops_max" : 333166,\n'
' "iops_mean" : '
'332391.785714,\n'
' "iops_stddev" : '
'468.114085,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.371429,\n'
' "sys_cpu" : 82.142857,\n'
' "ctx" : 13656,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.302774,\n'
' "10" : 82.745681,\n'
' "20" : 15.383217,\n'
' "50" : 1.505831,\n'
' "100" : 0.039759,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=141577: Wed Oct 26 04:38:05 '
'2022\n'
' read: IOPS=332k, BW=162MiB/s '
'(170MB/s)(1136MiB/7001msec)\n'
' slat (nsec): min=1142, '
'max=25858, avg=2197.27, '
'stdev=1099.93\n'
' clat (nsec): min=80, '
'max=165000, avg=9370.23, '
'stdev=2562.39\n'
' lat (usec): min=5, max=166, '
'avg=11.57, stdev= 2.63\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6048], '
'5.00th=[ 7648], 10.00th=[ 8640], '
'20.00th=[ 8768],\n'
' | 30.00th=[ 8768], '
'40.00th=[ 8768], 50.00th=[ '
'8896], 60.00th=[ 8896],\n'
' | 70.00th=[ 9024], '
'80.00th=[ 9152], '
'90.00th=[10304], '
'95.00th=[11584],\n'
' | 99.00th=[21888], '
'99.50th=[24192], '
'99.90th=[28800], '
'99.95th=[34048],\n'
' | 99.99th=[70144]\n'
' bw ( KiB/s): min=165820, '
'max=166583, per=100.00%, '
'avg=166195.86, stdev=234.10, '
'samples=14\n'
' iops : min=331640, '
'max=333166, avg=332391.79, '
'stdev=468.11, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.30%, 10=82.75%, 20=15.38%, '
'50=1.51%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=15.37%, '
'sys=82.14%, ctx=13656, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2326490,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=162MiB/s (170MB/s), '
'162MiB/s-162MiB/s '
'(170MB/s-170MB/s), io=1136MiB '
'(1191MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.output'),
'state': {'begin': 1666744796.147,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7930381298065186,
'end': 1666744796.9400382,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.state')},
'cmd_202': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.output'),
'state': {'begin': 1666744797.928816,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8015997409820557,
'end': 1666744798.7304158,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.state')},
'cmd_203': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.output'),
'state': {'begin': 1666744798.7327921,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.038516759872437,
'end': 1666744809.771309,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.state')},
'cmd_204': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751898,\n'
' "timestamp_ms" : '
'1666751898959,\n'
' "time" : "Wed Oct 26 04:38:18 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1195248640,\n'
' "io_kbytes" : 1167235,\n'
' "bw_bytes" : 170725416,\n'
' "bw" : 166724,\n'
' "iops" : 333447.507499,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2334466,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1142,\n'
' "max" : 21070,\n'
' "mean" : 2160.286033,\n'
' "stddev" : '
'1053.646322,\n'
' "N" : 2334467\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 70,\n'
' "max" : 172754,\n'
' "mean" : 9375.272461,\n'
' "stddev" : '
'2567.897705,\n'
' "N" : 2334469,\n'
' "percentile" : {\n'
' "1.000000" : 6048,\n'
' "5.000000" : 7584,\n'
' "10.000000" : 8640,\n'
' "20.000000" : 8768,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 8768,\n'
' "50.000000" : 8896,\n'
' "60.000000" : 8896,\n'
' "70.000000" : 8896,\n'
' "80.000000" : '
'10048,\n'
' "90.000000" : '
'10304,\n'
' "95.000000" : '
'11584,\n'
' "99.000000" : '
'21888,\n'
' "99.500000" : '
'23936,\n'
' "99.900000" : '
'28288,\n'
' "99.950000" : '
'33536,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5440,\n'
' "max" : 173937,\n'
' "mean" : '
'11535.557651,\n'
' "stddev" : '
'2590.223019,\n'
' "N" : 2334469\n'
' },\n'
' "bw_min" : 166368,\n'
' "bw_max" : 167296,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'166781.142857,\n'
' "bw_dev" : 248.585493,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 332737,\n'
' "iops_max" : 334592,\n'
' "iops_mean" : '
'333562.357143,\n'
' "iops_stddev" : '
'497.043197,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.000000,\n'
' "sys_cpu" : 82.514286,\n'
' "ctx" : 13822,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.012894,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.289659,\n'
' "10" : 79.779187,\n'
' "20" : 18.447945,\n'
' "50" : 1.422681,\n'
' "100" : 0.036282,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=141615: Wed Oct 26 04:38:18 '
'2022\n'
' read: IOPS=333k, BW=163MiB/s '
'(171MB/s)(1140MiB/7001msec)\n'
' slat (nsec): min=1142, '
'max=21070, avg=2160.29, '
'stdev=1053.65\n'
' clat (nsec): min=70, '
'max=172754, avg=9375.27, '
'stdev=2567.90\n'
' lat (usec): min=5, max=173, '
'avg=11.54, stdev= 2.59\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6048], '
'5.00th=[ 7584], 10.00th=[ 8640], '
'20.00th=[ 8768],\n'
' | 30.00th=[ 8768], '
'40.00th=[ 8768], 50.00th=[ '
'8896], 60.00th=[ 8896],\n'
' | 70.00th=[ 8896], '
'80.00th=[10048], '
'90.00th=[10304], '
'95.00th=[11584],\n'
' | 99.00th=[21888], '
'99.50th=[23936], '
'99.90th=[28288], '
'99.95th=[33536],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=166368, '
'max=167296, per=100.00%, '
'avg=166781.14, stdev=248.59, '
'samples=14\n'
' iops : min=332737, '
'max=334592, avg=333562.36, '
'stdev=497.04, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.29%, 10=79.78%, 20=18.45%, '
'50=1.42%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=15.00%, '
'sys=82.51%, ctx=13822, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2334466,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=163MiB/s (171MB/s), '
'163MiB/s-163MiB/s '
'(171MB/s-171MB/s), io=1140MiB '
'(1195MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.output'),
'state': {'begin': 1666744809.7744377,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7913088798522949,
'end': 1666744810.5657465,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.state')},
'cmd_205': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.output'),
'state': {'begin': 1666744811.5587492,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7833278179168701,
'end': 1666744812.342077,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.state')},
'cmd_206': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.output'),
'state': {'begin': 1666744812.344384,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.040228605270386,
'end': 1666744823.3846126,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.state')},
'cmd_207': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751912,\n'
' "timestamp_ms" : '
'1666751912572,\n'
' "time" : "Wed Oct 26 04:38:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1193305600,\n'
' "io_kbytes" : 1165337,\n'
' "bw_bytes" : 170447878,\n'
' "bw" : 166453,\n'
' "iops" : 332905.442080,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2330671,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1142,\n'
' "max" : 25989,\n'
' "mean" : 2188.620048,\n'
' "stddev" : '
'1059.038506,\n'
' "N" : 2330672\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 70,\n'
' "max" : 194865,\n'
' "mean" : 9353.569892,\n'
' "stddev" : '
'2502.558563,\n'
' "N" : 2330674,\n'
' "percentile" : {\n'
' "1.000000" : 6048,\n'
' "5.000000" : 7584,\n'
' "10.000000" : 8640,\n'
' "20.000000" : 8768,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 8768,\n'
' "50.000000" : 8896,\n'
' "60.000000" : 8896,\n'
' "70.000000" : 8896,\n'
' "80.000000" : 9280,\n'
' "90.000000" : '
'10304,\n'
' "95.000000" : '
'11584,\n'
' "99.000000" : '
'21888,\n'
' "99.500000" : '
'23680,\n'
' "99.900000" : '
'27264,\n'
' "99.950000" : '
'32640,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5451,\n'
' "max" : 196047,\n'
' "mean" : '
'11542.189115,\n'
' "stddev" : '
'2537.835066,\n'
' "N" : 2330674\n'
' },\n'
' "bw_min" : 166200,\n'
' "bw_max" : 167348,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'166499.928571,\n'
' "bw_dev" : 288.296713,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 332400,\n'
' "iops_max" : 334696,\n'
' "iops_mean" : '
'332999.857143,\n'
' "iops_stddev" : '
'576.593425,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.900000,\n'
' "sys_cpu" : 83.671429,\n'
' "ctx" : 13746,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.012014,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.293006,\n'
' "10" : 81.953008,\n'
' "20" : 16.269478,\n'
' "50" : 1.433021,\n'
' "100" : 0.029820,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=141653: Wed Oct 26 04:38:32 '
'2022\n'
' read: IOPS=333k, BW=163MiB/s '
'(170MB/s)(1138MiB/7001msec)\n'
' slat (nsec): min=1142, '
'max=25989, avg=2188.62, '
'stdev=1059.04\n'
' clat (nsec): min=70, '
'max=194865, avg=9353.57, '
'stdev=2502.56\n'
' lat (usec): min=5, max=196, '
'avg=11.54, stdev= 2.54\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6048], '
'5.00th=[ 7584], 10.00th=[ 8640], '
'20.00th=[ 8768],\n'
' | 30.00th=[ 8768], '
'40.00th=[ 8768], 50.00th=[ '
'8896], 60.00th=[ 8896],\n'
' | 70.00th=[ 8896], '
'80.00th=[ 9280], '
'90.00th=[10304], '
'95.00th=[11584],\n'
' | 99.00th=[21888], '
'99.50th=[23680], '
'99.90th=[27264], '
'99.95th=[32640],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=166200, '
'max=167348, per=100.00%, '
'avg=166499.93, stdev=288.30, '
'samples=14\n'
' iops : min=332400, '
'max=334696, avg=332999.86, '
'stdev=576.59, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.29%, 10=81.95%, 20=16.27%, '
'50=1.43%\n'
' lat (usec) : 100=0.03%, '
'250=0.01%\n'
' cpu : usr=13.90%, '
'sys=83.67%, ctx=13746, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2330671,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=163MiB/s (170MB/s), '
'163MiB/s-163MiB/s '
'(170MB/s-170MB/s), io=1138MiB '
'(1193MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.output'),
'state': {'begin': 1666744823.3879497,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.786017894744873,
'end': 1666744824.1739676,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.state')},
'cmd_208': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.output'),
'state': {'begin': 1666744825.1965132,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7827303409576416,
'end': 1666744825.9792435,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.state')},
'cmd_209': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.output'),
'state': {'begin': 1666744825.9816153,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.062126398086548,
'end': 1666744837.0437417,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.state')},
'cmd_21': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751023,\n'
' "timestamp_ms" : '
'1666751023343,\n'
' "time" : "Wed Oct 26 04:23:43 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 477709312,\n'
' "io_kbytes" : 466513,\n'
' "bw_bytes" : 68234439,\n'
' "bw" : 66635,\n'
' "iops" : 133270.389944,\n'
' "runtime" : 7001,\n'
' "total_ios" : 933026,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 250,\n'
' "max" : 10320,\n'
' "mean" : 279.075861,\n'
' "stddev" : 23.844744,\n'
' "N" : 933026\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 250,\n'
' "max" : 119153,\n'
' "mean" : 7007.570928,\n'
' "stddev" : '
'1331.921014,\n'
' "N" : 933026,\n'
' "percentile" : {\n'
' "1.000000" : 6624,\n'
' "5.000000" : 6688,\n'
' "10.000000" : 6752,\n'
' "20.000000" : 6752,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 6816,\n'
' "50.000000" : 6816,\n'
' "60.000000" : 6880,\n'
' "70.000000" : 6880,\n'
' "80.000000" : 6944,\n'
' "90.000000" : 7008,\n'
' "95.000000" : 7136,\n'
' "99.000000" : 14400,\n'
' "99.500000" : 17024,\n'
' "99.900000" : 18048,\n'
' "99.950000" : 18560,\n'
' "99.990000" : 59136\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6392,\n'
' "max" : 119454,\n'
' "mean" : 7286.646789,\n'
' "stddev" : '
'1332.043757,\n'
' "N" : 933026\n'
' },\n'
' "bw_min" : 66542,\n'
' "bw_max" : 66739,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'66649.214286,\n'
' "bw_dev" : 62.060986,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 133084,\n'
' "iops_max" : 133478,\n'
' "iops_mean" : '
'133298.428571,\n'
' "iops_stddev" : '
'124.121971,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 27,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.287508,\n'
' "20" : 1.672622,\n'
' "50" : 0.025401,\n'
' "100" : 0.014148,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=138918: Wed Oct 26 04:23:43 '
'2022\n'
' read: IOPS=133k, BW=65.1MiB/s '
'(68.2MB/s)(456MiB/7001msec)\n'
' slat (nsec): min=250, '
'max=10320, avg=279.08, '
'stdev=23.84\n'
' clat (nsec): min=250, '
'max=119153, avg=7007.57, '
'stdev=1331.92\n'
' lat (usec): min=6, max=119, '
'avg= 7.29, stdev= 1.33\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6624], 5.00th=[ '
'6688], 10.00th=[ 6752], 20.00th=[ '
'6752],\n'
' | 30.00th=[ 6752], 40.00th=[ '
'6816], 50.00th=[ 6816], 60.00th=[ '
'6880],\n'
' | 70.00th=[ 6880], 80.00th=[ '
'6944], 90.00th=[ 7008], 95.00th=[ '
'7136],\n'
' | 99.00th=[14400], '
'99.50th=[17024], 99.90th=[18048], '
'99.95th=[18560],\n'
' | 99.99th=[59136]\n'
' bw ( KiB/s): min=66542, '
'max=66739, per=100.00%, '
'avg=66649.21, stdev=62.06, '
'samples=14\n'
' iops : min=133084, '
'max=133478, avg=133298.43, '
'stdev=124.12, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=98.29%, '
'20=1.67%, 50=0.03%, 100=0.01%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=27, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=933026,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=65.1MiB/s (68.2MB/s), '
'65.1MiB/s-65.1MiB/s '
'(68.2MB/s-68.2MB/s), io=456MiB '
'(478MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.output'),
'state': {'begin': 1666743934.213952,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7880222797393799,
'end': 1666743935.0019743,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.state')},
'cmd_210': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751926,\n'
' "timestamp_ms" : '
'1666751926230,\n'
' "time" : "Wed Oct 26 04:38:46 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1147879936,\n'
' "io_kbytes" : 1120976,\n'
' "bw_bytes" : 163959425,\n'
' "bw" : 160116,\n'
' "iops" : 320232.681046,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2241949,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1193,\n'
' "max" : 33182,\n'
' "mean" : 2542.839603,\n'
' "stddev" : '
'751.011231,\n'
' "N" : 2241950\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 120,\n'
' "max" : 161323,\n'
' "mean" : 9665.963411,\n'
' "stddev" : '
'2557.083231,\n'
' "N" : 2241952,\n'
' "percentile" : {\n'
' "1.000000" : 6368,\n'
' "5.000000" : 8032,\n'
' "10.000000" : 9024,\n'
' "20.000000" : 9024,\n'
' "30.000000" : 9152,\n'
' "40.000000" : 9152,\n'
' "50.000000" : 9280,\n'
' "60.000000" : 9280,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9408,\n'
' "90.000000" : '
'10688,\n'
' "95.000000" : '
'10944,\n'
' "99.000000" : '
'22912,\n'
' "99.500000" : '
'24704,\n'
' "99.900000" : '
'28800,\n'
' "99.950000" : '
'33536,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5630,\n'
' "max" : 162565,\n'
' "mean" : '
'12208.801880,\n'
' "stddev" : '
'2451.667220,\n'
' "N" : 2241952\n'
' },\n'
' "bw_min" : 159647,\n'
' "bw_max" : 160652,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'160159.214286,\n'
' "bw_dev" : 275.571065,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 319294,\n'
' "iops_max" : 321304,\n'
' "iops_mean" : '
'320318.500000,\n'
' "iops_stddev" : '
'551.411999,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.742857,\n'
' "sys_cpu" : 78.857143,\n'
' "ctx" : 12881,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.304556,\n'
' "10" : 85.096405,\n'
' "20" : 12.938698,\n'
' "50" : 1.610117,\n'
' "100" : 0.036888,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141691: Wed '
'Oct 26 04:38:46 2022\n'
' read: IOPS=320k, BW=156MiB/s '
'(164MB/s)(1095MiB/7001msec)\n'
' slat (nsec): min=1193, '
'max=33182, avg=2542.84, '
'stdev=751.01\n'
' clat (nsec): min=120, '
'max=161323, avg=9665.96, '
'stdev=2557.08\n'
' lat (usec): min=5, max=162, '
'avg=12.21, stdev= 2.45\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6368], '
'5.00th=[ 8032], 10.00th=[ 9024], '
'20.00th=[ 9024],\n'
' | 30.00th=[ 9152], '
'40.00th=[ 9152], 50.00th=[ '
'9280], 60.00th=[ 9280],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9408], '
'90.00th=[10688], '
'95.00th=[10944],\n'
' | 99.00th=[22912], '
'99.50th=[24704], '
'99.90th=[28800], '
'99.95th=[33536],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=159647, '
'max=160652, per=100.00%, '
'avg=160159.21, stdev=275.57, '
'samples=14\n'
' iops : min=319294, '
'max=321304, avg=320318.50, '
'stdev=551.41, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.30%, 10=85.10%, 20=12.94%, '
'50=1.61%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=18.74%, '
'sys=78.86%, ctx=12881, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2241949,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=156MiB/s (164MB/s), '
'156MiB/s-156MiB/s '
'(164MB/s-164MB/s), io=1095MiB '
'(1148MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.output'),
'state': {'begin': 1666744837.0469315,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.792938232421875,
'end': 1666744837.8398697,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.state')},
'cmd_211': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.output'),
'state': {'begin': 1666744838.8321114,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7809031009674072,
'end': 1666744839.6130145,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.state')},
'cmd_212': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.output'),
'state': {'begin': 1666744839.6151216,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.057931184768677,
'end': 1666744850.6730528,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.state')},
'cmd_213': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751939,\n'
' "timestamp_ms" : '
'1666751939860,\n'
' "time" : "Wed Oct 26 04:38:59 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1146219008,\n'
' "io_kbytes" : 1119354,\n'
' "bw_bytes" : 163722183,\n'
' "bw" : 159884,\n'
' "iops" : 319769.318669,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2238705,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1202,\n'
' "max" : 26811,\n'
' "mean" : 2540.398739,\n'
' "stddev" : '
'740.364198,\n'
' "N" : 2238706\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 110,\n'
' "max" : 124784,\n'
' "mean" : 9685.766814,\n'
' "stddev" : '
'2569.003445,\n'
' "N" : 2238708,\n'
' "percentile" : {\n'
' "1.000000" : 6432,\n'
' "5.000000" : 8096,\n'
' "10.000000" : 9024,\n'
' "20.000000" : 9024,\n'
' "30.000000" : 9152,\n'
' "40.000000" : 9152,\n'
' "50.000000" : 9280,\n'
' "60.000000" : 9280,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9408,\n'
' "90.000000" : '
'10688,\n'
' "95.000000" : '
'10944,\n'
' "99.000000" : '
'23168,\n'
' "99.500000" : '
'24704,\n'
' "99.900000" : '
'29568,\n'
' "99.950000" : '
'34048,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5580,\n'
' "max" : 126016,\n'
' "mean" : '
'12226.165602,\n'
' "stddev" : '
'2465.670806,\n'
' "N" : 2238708\n'
' },\n'
' "bw_min" : 159323,\n'
' "bw_max" : 160812,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'159930.214286,\n'
' "bw_dev" : 504.450833,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 318646,\n'
' "iops_max" : 321625,\n'
' "iops_mean" : '
'319860.500000,\n'
' "iops_stddev" : '
'1009.131595,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.500000,\n'
' "sys_cpu" : 82.100000,\n'
' "ctx" : 12883,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.293429,\n'
' "10" : 84.859327,\n'
' "20" : 13.183113,\n'
' "50" : 1.610619,\n'
' "100" : 0.040336,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141729: Wed '
'Oct 26 04:38:59 2022\n'
' read: IOPS=320k, BW=156MiB/s '
'(164MB/s)(1093MiB/7001msec)\n'
' slat (nsec): min=1202, '
'max=26811, avg=2540.40, '
'stdev=740.36\n'
' clat (nsec): min=110, '
'max=124784, avg=9685.77, '
'stdev=2569.00\n'
' lat (usec): min=5, max=126, '
'avg=12.23, stdev= 2.47\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6432], '
'5.00th=[ 8096], 10.00th=[ 9024], '
'20.00th=[ 9024],\n'
' | 30.00th=[ 9152], '
'40.00th=[ 9152], 50.00th=[ '
'9280], 60.00th=[ 9280],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9408], '
'90.00th=[10688], '
'95.00th=[10944],\n'
' | 99.00th=[23168], '
'99.50th=[24704], '
'99.90th=[29568], '
'99.95th=[34048],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=159323, '
'max=160812, per=100.00%, '
'avg=159930.21, stdev=504.45, '
'samples=14\n'
' iops : min=318646, '
'max=321625, avg=319860.50, '
'stdev=1009.13, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.29%, 10=84.86%, 20=13.18%, '
'50=1.61%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=15.50%, '
'sys=82.10%, ctx=12883, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2238705,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=156MiB/s (164MB/s), '
'156MiB/s-156MiB/s '
'(164MB/s-164MB/s), io=1093MiB '
'(1146MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.output'),
'state': {'begin': 1666744850.6764238,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7867345809936523,
'end': 1666744851.4631584,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.state')},
'cmd_214': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.output'),
'state': {'begin': 1666744852.4581199,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7854704856872559,
'end': 1666744853.2435904,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.state')},
'cmd_215': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.output'),
'state': {'begin': 1666744853.245876,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.065338611602783,
'end': 1666744864.3112147,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.state')},
'cmd_216': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751953,\n'
' "timestamp_ms" : '
'1666751953498,\n'
' "time" : "Wed Oct 26 04:39:13 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1122532864,\n'
' "io_kbytes" : 1096223,\n'
' "bw_bytes" : 160338932,\n'
' "bw" : 156580,\n'
' "iops" : 313161.405513,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2192443,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1212,\n'
' "max" : 24466,\n'
' "mean" : 2454.161673,\n'
' "stddev" : '
'961.853907,\n'
' "N" : 2192444\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 130,\n'
' "max" : 143108,\n'
' "mean" : 9983.846668,\n'
' "stddev" : '
'2529.342162,\n'
' "N" : 2192446,\n'
' "percentile" : {\n'
' "1.000000" : 7648,\n'
' "5.000000" : 8160,\n'
' "10.000000" : 9280,\n'
' "20.000000" : 9280,\n'
' "30.000000" : 9408,\n'
' "40.000000" : 9408,\n'
' "50.000000" : 9536,\n'
' "60.000000" : 9536,\n'
' "70.000000" : 9536,\n'
' "80.000000" : '
'10688,\n'
' "90.000000" : '
'10944,\n'
' "95.000000" : '
'11072,\n'
' "99.000000" : '
'22912,\n'
' "99.500000" : '
'24704,\n'
' "99.900000" : '
'28544,\n'
' "99.950000" : '
'33536,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5500,\n'
' "max" : 144361,\n'
' "mean" : '
'12438.008588,\n'
' "stddev" : '
'2576.508476,\n'
' "N" : 2192446\n'
' },\n'
' "bw_min" : 156329,\n'
' "bw_max" : 156855,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'156627.857143,\n'
' "bw_dev" : 180.425078,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 312658,\n'
' "iops_max" : 313710,\n'
' "iops_mean" : '
'313255.714286,\n'
' "iops_stddev" : '
'361.150177,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 20.500000,\n'
' "sys_cpu" : 77.171429,\n'
' "ctx" : 12638,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.095647,\n'
' "10" : 78.931676,\n'
' "20" : 19.299658,\n'
' "50" : 1.626907,\n'
' "100" : 0.036307,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=141767: Wed '
'Oct 26 04:39:13 2022\n'
' read: IOPS=313k, BW=153MiB/s '
'(160MB/s)(1071MiB/7001msec)\n'
' slat (nsec): min=1212, '
'max=24466, avg=2454.16, '
'stdev=961.85\n'
' clat (nsec): min=130, '
'max=143108, avg=9983.85, '
'stdev=2529.34\n'
' lat (usec): min=5, max=144, '
'avg=12.44, stdev= 2.58\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7648], '
'5.00th=[ 8160], 10.00th=[ 9280], '
'20.00th=[ 9280],\n'
' | 30.00th=[ 9408], '
'40.00th=[ 9408], 50.00th=[ '
'9536], 60.00th=[ 9536],\n'
' | 70.00th=[ 9536], '
'80.00th=[10688], '
'90.00th=[10944], '
'95.00th=[11072],\n'
' | 99.00th=[22912], '
'99.50th=[24704], '
'99.90th=[28544], '
'99.95th=[33536],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=156329, '
'max=156855, per=100.00%, '
'avg=156627.86, stdev=180.43, '
'samples=14\n'
' iops : min=312658, '
'max=313710, avg=313255.71, '
'stdev=361.15, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.10%, 10=78.93%, 20=19.30%, '
'50=1.63%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=20.50%, '
'sys=77.17%, ctx=12638, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2192443,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=153MiB/s (160MB/s), '
'153MiB/s-153MiB/s '
'(160MB/s-160MB/s), io=1071MiB '
'(1123MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.output'),
'state': {'begin': 1666744864.3145394,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7896699905395508,
'end': 1666744865.1042094,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.state')},
'cmd_217': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.output'),
'state': {'begin': 1666744867.1236086,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7923183441162109,
'end': 1666744867.915927,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.state')},
'cmd_218': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.output'),
'state': {'begin': 1666744867.9183056,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.49340534210205,
'end': 1666744879.411711,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.state')},
'cmd_219': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751968,\n'
' "timestamp_ms" : '
'1666751968527,\n'
' "time" : "Wed Oct 26 04:39:28 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1505933312,\n'
' "io_kbytes" : 1470638,\n'
' "bw_bytes" : 215102601,\n'
' "bw" : 210061,\n'
' "iops" : 420121.839737,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2941273,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 210,\n'
' "max" : 28404,\n'
' "mean" : 272.227385,\n'
' "stddev" : 44.424831,\n'
' "N" : 2941273\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 360,\n'
' "max" : 125015,\n'
' "mean" : 9017.813720,\n'
' "stddev" : '
'2595.881437,\n'
' "N" : 2941276,\n'
' "percentile" : {\n'
' "1.000000" : 6752,\n'
' "5.000000" : 7456,\n'
' "10.000000" : 7840,\n'
' "20.000000" : 7968,\n'
' "30.000000" : 8032,\n'
' "40.000000" : 8160,\n'
' "50.000000" : 8256,\n'
' "60.000000" : 9152,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9664,\n'
' "95.000000" : '
'10560,\n'
' "99.000000" : '
'20352,\n'
' "99.500000" : '
'22144,\n'
' "99.900000" : '
'29568,\n'
' "99.950000" : '
'36096,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5751,\n'
' "max" : 125285,\n'
' "mean" : 9290.041100,\n'
' "stddev" : '
'2596.346238,\n'
' "N" : 2941276\n'
' },\n'
' "bw_min" : 209329,\n'
' "bw_max" : 211028,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'210143.428571,\n'
' "bw_dev" : 467.370338,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 418658,\n'
' "iops_max" : 422056,\n'
' "iops_mean" : '
'420286.428571,\n'
' "iops_stddev" : '
'934.806662,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 21,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 1.919441,\n'
' "4" : 98.072977,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 93.588456,\n'
' "20" : 5.184796,\n'
' "50" : 1.186323,\n'
' "100" : 0.035801,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=141818: Wed Oct 26 04:39:28 '
'2022\n'
' read: IOPS=420k, BW=205MiB/s '
'(215MB/s)(1436MiB/7001msec)\n'
' slat (nsec): min=210, '
'max=28404, avg=272.23, '
'stdev=44.42\n'
' clat (nsec): min=360, '
'max=125015, avg=9017.81, '
'stdev=2595.88\n'
' lat (usec): min=5, max=125, '
'avg= 9.29, stdev= 2.60\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6752], '
'5.00th=[ 7456], 10.00th=[ 7840], '
'20.00th=[ 7968],\n'
' | 30.00th=[ 8032], '
'40.00th=[ 8160], 50.00th=[ '
'8256], 60.00th=[ 9152],\n'
' | 70.00th=[ 9280], '
'80.00th=[ 9408], 90.00th=[ '
'9664], 95.00th=[10560],\n'
' | 99.00th=[20352], '
'99.50th=[22144], '
'99.90th=[29568], '
'99.95th=[36096],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=209329, '
'max=211028, per=100.00%, '
'avg=210143.43, stdev=467.37, '
'samples=14\n'
' iops : min=418658, '
'max=422056, avg=420286.43, '
'stdev=934.81, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=93.59%, 20=5.18%, '
'50=1.19%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=21, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=1.9%, '
'4=98.1%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2941273,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=205MiB/s (215MB/s), '
'205MiB/s-205MiB/s '
'(215MB/s-215MB/s), io=1436MiB '
'(1506MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.output'),
'state': {'begin': 1666744879.414543,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7872753143310547,
'end': 1666744880.2018182,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.state')},
'cmd_22': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.output'),
'state': {'begin': 1666743937.0274687,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7875986099243164,
'end': 1666743937.8150673,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.state')},
'cmd_220': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.output'),
'state': {'begin': 1666744882.2170968,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7816946506500244,
'end': 1666744882.9987915,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.state')},
'cmd_221': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.output'),
'state': {'begin': 1666744883.0009136,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.48667311668396,
'end': 1666744894.4875867,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.state')},
'cmd_222': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751983,\n'
' "timestamp_ms" : '
'1666751983603,\n'
' "time" : "Wed Oct 26 04:39:43 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1417665024,\n'
' "io_kbytes" : 1384438,\n'
' "bw_bytes" : 202494647,\n'
' "bw" : 197748,\n'
' "iops" : 395496.929010,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2768874,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 170,\n'
' "max" : 8436,\n'
' "mean" : 264.949002,\n'
' "stddev" : 22.607211,\n'
' "N" : 2768874\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2355,\n'
' "max" : 122710,\n'
' "mean" : 9554.498064,\n'
' "stddev" : '
'2412.332312,\n'
' "N" : 2768877,\n'
' "percentile" : {\n'
' "1.000000" : 6432,\n'
' "5.000000" : 7392,\n'
' "10.000000" : 8032,\n'
' "20.000000" : 8768,\n'
' "30.000000" : 9152,\n'
' "40.000000" : 9152,\n'
' "50.000000" : 9280,\n'
' "60.000000" : 9408,\n'
' "70.000000" : 9536,\n'
' "80.000000" : 9792,\n'
' "90.000000" : '
'10432,\n'
' "95.000000" : '
'11840,\n'
' "99.000000" : '
'20096,\n'
' "99.500000" : '
'22144,\n'
' "99.900000" : '
'28032,\n'
' "99.950000" : '
'35072,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5390,\n'
' "max" : 122971,\n'
' "mean" : 9819.447028,\n'
' "stddev" : '
'2413.593380,\n'
' "N" : 2768877\n'
' },\n'
' "bw_min" : 197589,\n'
' "bw_max" : 198348,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'197841.000000,\n'
' "bw_dev" : 217.461933,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 395178,\n'
' "iops_max" : 396696,\n'
' "iops_mean" : '
'395682.071429,\n'
' "iops_stddev" : '
'435.142499,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 24,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 11.586082,\n'
' "4" : 88.411318,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 87.145894,\n'
' "20" : 11.771283,\n'
' "50" : 1.041795,\n'
' "100" : 0.036007,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=141869: Wed Oct 26 04:39:43 '
'2022\n'
' read: IOPS=395k, BW=193MiB/s '
'(202MB/s)(1352MiB/7001msec)\n'
' slat (nsec): min=170, '
'max=8436, avg=264.95, '
'stdev=22.61\n'
' clat (usec): min=2, max=122, '
'avg= 9.55, stdev= 2.41\n'
' lat (usec): min=5, max=122, '
'avg= 9.82, stdev= 2.41\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6432], '
'5.00th=[ 7392], 10.00th=[ 8032], '
'20.00th=[ 8768],\n'
' | 30.00th=[ 9152], '
'40.00th=[ 9152], 50.00th=[ '
'9280], 60.00th=[ 9408],\n'
' | 70.00th=[ 9536], '
'80.00th=[ 9792], '
'90.00th=[10432], '
'95.00th=[11840],\n'
' | 99.00th=[20096], '
'99.50th=[22144], '
'99.90th=[28032], '
'99.95th=[35072],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=197589, '
'max=198348, per=100.00%, '
'avg=197841.00, stdev=217.46, '
'samples=14\n'
' iops : min=395178, '
'max=396696, avg=395682.07, '
'stdev=435.14, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=87.15%, 20=11.77%, 50=1.04%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=24, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, '
'2=11.6%, 4=88.4%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2768874,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=193MiB/s (202MB/s), '
'193MiB/s-193MiB/s '
'(202MB/s-202MB/s), io=1352MiB '
'(1418MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.output'),
'state': {'begin': 1666744894.4904203,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.791827917098999,
'end': 1666744895.2822483,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.state')},
'cmd_223': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.output'),
'state': {'begin': 1666744897.3290775,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7823565006256104,
'end': 1666744898.111434,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.state')},
'cmd_224': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.output'),
'state': {'begin': 1666744898.1135473,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.458956718444824,
'end': 1666744909.572504,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.state')},
'cmd_225': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751998,\n'
' "timestamp_ms" : '
'1666751998687,\n'
' "time" : "Wed Oct 26 04:39:58 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1425394688,\n'
' "io_kbytes" : 1391987,\n'
' "bw_bytes" : 203598727,\n'
' "bw" : 198826,\n'
' "iops" : 397653.335238,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2783971,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 180,\n'
' "max" : 14417,\n'
' "mean" : 263.192554,\n'
' "stddev" : 26.016764,\n'
' "N" : 2783971\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 391,\n'
' "max" : 156223,\n'
' "mean" : 9514.911947,\n'
' "stddev" : '
'2418.859053,\n'
' "N" : 2783974,\n'
' "percentile" : {\n'
' "1.000000" : 6432,\n'
' "5.000000" : 7456,\n'
' "10.000000" : 8160,\n'
' "20.000000" : 8768,\n'
' "30.000000" : 9024,\n'
' "40.000000" : 9152,\n'
' "50.000000" : 9280,\n'
' "60.000000" : 9280,\n'
' "70.000000" : 9408,\n'
' "80.000000" : 9536,\n'
' "90.000000" : '
'10176,\n'
' "95.000000" : '
'11584,\n'
' "99.000000" : '
'20352,\n'
' "99.500000" : '
'22144,\n'
' "99.900000" : '
'27776,\n'
' "99.950000" : '
'34560,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5100,\n'
' "max" : 156484,\n'
' "mean" : 9778.104498,\n'
' "stddev" : '
'2419.649924,\n'
' "N" : 2783974\n'
' },\n'
' "bw_min" : 198533,\n'
' "bw_max" : 199591,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'198952.285714,\n'
' "bw_dev" : 317.459247,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 397066,\n'
' "iops_max" : 399182,\n'
' "iops_mean" : '
'397904.714286,\n'
' "iops_stddev" : '
'634.979519,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 16,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 7.956728,\n'
' "4" : 92.041656,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 88.694997,\n'
' "20" : 10.182757,\n'
' "50" : 1.083416,\n'
' "100" : 0.033872,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=141919: Wed Oct 26 04:39:58 '
'2022\n'
' read: IOPS=398k, BW=194MiB/s '
'(204MB/s)(1359MiB/7001msec)\n'
' slat (nsec): min=180, '
'max=14417, avg=263.19, '
'stdev=26.02\n'
' clat (nsec): min=391, '
'max=156223, avg=9514.91, '
'stdev=2418.86\n'
' lat (usec): min=5, max=156, '
'avg= 9.78, stdev= 2.42\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6432], '
'5.00th=[ 7456], 10.00th=[ 8160], '
'20.00th=[ 8768],\n'
' | 30.00th=[ 9024], '
'40.00th=[ 9152], 50.00th=[ '
'9280], 60.00th=[ 9280],\n'
' | 70.00th=[ 9408], '
'80.00th=[ 9536], '
'90.00th=[10176], '
'95.00th=[11584],\n'
' | 99.00th=[20352], '
'99.50th=[22144], '
'99.90th=[27776], '
'99.95th=[34560],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=198533, '
'max=199591, per=100.00%, '
'avg=198952.29, stdev=317.46, '
'samples=14\n'
' iops : min=397066, '
'max=399182, avg=397904.71, '
'stdev=634.98, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=88.69%, 20=10.18%, 50=1.08%, '
'100=0.03%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=16, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=8.0%, '
'4=92.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2783971,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=194MiB/s (204MB/s), '
'194MiB/s-194MiB/s '
'(204MB/s-204MB/s), io=1359MiB '
'(1425MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.output'),
'state': {'begin': 1666744909.5753038,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7929620742797852,
'end': 1666744910.3682659,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.state')},
'cmd_226': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.output'),
'state': {'begin': 1666744911.3604612,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.787114143371582,
'end': 1666744912.1475754,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.state')},
'cmd_227': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.output'),
'state': {'begin': 1666744912.149671,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.040347814559937,
'end': 1666744923.190019,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.state')},
'cmd_228': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752012,\n'
' "timestamp_ms" : '
'1666752012376,\n'
' "time" : "Wed Oct 26 04:40:12 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1110281216,\n'
' "io_kbytes" : 1084259,\n'
' "bw_bytes" : 158588946,\n'
' "bw" : 154872,\n'
' "iops" : 309743.465219,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2168514,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1292,\n'
' "max" : 24085,\n'
' "mean" : 2504.110085,\n'
' "stddev" : '
'952.888239,\n'
' "N" : 2168515\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 198122,\n'
' "mean" : '
'10032.524192,\n'
' "stddev" : '
'2637.425346,\n'
' "N" : 2168517,\n'
' "percentile" : {\n'
' "1.000000" : 7520,\n'
' "5.000000" : 8032,\n'
' "10.000000" : 9280,\n'
' "20.000000" : 9408,\n'
' "30.000000" : 9408,\n'
' "40.000000" : 9408,\n'
' "50.000000" : 9536,\n'
' "60.000000" : 9536,\n'
' "70.000000" : 9664,\n'
' "80.000000" : '
'10688,\n'
' "90.000000" : '
'11072,\n'
' "95.000000" : '
'11328,\n'
' "99.000000" : '
'23168,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'29312,\n'
' "99.950000" : '
'34560,\n'
' "99.990000" : 73216\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5731,\n'
' "max" : 199454,\n'
' "mean" : '
'12536.634481,\n'
' "stddev" : '
'2666.909686,\n'
' "N" : 2168517\n'
' },\n'
' "bw_min" : 154763,\n'
' "bw_max" : 155474,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'154943.461538,\n'
' "bw_dev" : 227.947953,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 309526,\n'
' "iops_max" : 310949,\n'
' "iops_mean" : '
'309887.000000,\n'
' "iops_stddev" : '
'456.089903,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.742857,\n'
' "sys_cpu" : 82.900000,\n'
' "ctx" : 12590,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.086188,\n'
' "10" : 78.649711,\n'
' "20" : 19.549194,\n'
' "50" : 1.657863,\n'
' "100" : 0.040166,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 3043416,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 15511,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 15511,\n'
' "util" : 99.035239\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=141958: Wed '
'Oct 26 04:40:12 2022\n'
' read: IOPS=310k, BW=151MiB/s '
'(159MB/s)(1059MiB/7001msec)\n'
' slat (nsec): min=1292, '
'max=24085, avg=2504.11, '
'stdev=952.89\n'
' clat (nsec): min=80, '
'max=198122, avg=10032.52, '
'stdev=2637.43\n'
' lat (usec): min=5, max=199, '
'avg=12.54, stdev= 2.67\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7520], '
'5.00th=[ 8032], 10.00th=[ 9280], '
'20.00th=[ 9408],\n'
' | 30.00th=[ 9408], '
'40.00th=[ 9408], 50.00th=[ '
'9536], 60.00th=[ 9536],\n'
' | 70.00th=[ 9664], '
'80.00th=[10688], '
'90.00th=[11072], '
'95.00th=[11328],\n'
' | 99.00th=[23168], '
'99.50th=[24960], '
'99.90th=[29312], '
'99.95th=[34560],\n'
' | 99.99th=[73216]\n'
' bw ( KiB/s): min=154763, '
'max=155474, per=100.00%, '
'avg=154943.46, stdev=227.95, '
'samples=13\n'
' iops : min=309526, '
'max=310949, avg=309887.00, '
'stdev=456.09, samples=13\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.09%, 10=78.65%, 20=19.55%, '
'50=1.66%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=14.74%, '
'sys=82.90%, ctx=12590, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2168514,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=151MiB/s (159MB/s), '
'151MiB/s-151MiB/s '
'(159MB/s-159MB/s), io=1059MiB '
'(1110MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=3043416/0, '
'merge=0/0, ticks=15511/0, '
'in_queue=15511, util=99.04%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.output'),
'state': {'begin': 1666744923.1928506,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7912318706512451,
'end': 1666744923.9840825,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.state')},
'cmd_229': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.output'),
'state': {'begin': 1666744924.9830532,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7857077121734619,
'end': 1666744925.768761,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.state')},
'cmd_23': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.output'),
'state': {'begin': 1666743937.821821,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.486093521118164,
'end': 1666743949.3079145,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.state')},
'cmd_230': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.output'),
'state': {'begin': 1666744925.7707062,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.034291982650757,
'end': 1666744936.8049982,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.state')},
'cmd_231': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752025,\n'
' "timestamp_ms" : '
'1666752025990,\n'
' "time" : "Wed Oct 26 04:40:25 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1115557888,\n'
' "io_kbytes" : 1089412,\n'
' "bw_bytes" : 159342649,\n'
' "bw" : 155608,\n'
' "iops" : 311215.540637,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2178820,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1292,\n'
' "max" : 28123,\n'
' "mean" : 2499.686404,\n'
' "stddev" : '
'889.896990,\n'
' "N" : 2178821\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 70,\n'
' "max" : 142838,\n'
' "mean" : 9952.938270,\n'
' "stddev" : '
'2613.804392,\n'
' "N" : 2178823,\n'
' "percentile" : {\n'
' "1.000000" : 7392,\n'
' "5.000000" : 8032,\n'
' "10.000000" : 9152,\n'
' "20.000000" : 9280,\n'
' "30.000000" : 9408,\n'
' "40.000000" : 9408,\n'
' "50.000000" : 9536,\n'
' "60.000000" : 9536,\n'
' "70.000000" : 9536,\n'
' "80.000000" : '
'10432,\n'
' "90.000000" : '
'11072,\n'
' "95.000000" : '
'11328,\n'
' "99.000000" : '
'23168,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'29824,\n'
' "99.950000" : '
'34560,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5541,\n'
' "max" : 144180,\n'
' "mean" : '
'12452.624922,\n'
' "stddev" : '
'2606.970828,\n'
' "N" : 2178823\n'
' },\n'
' "bw_min" : 155411,\n'
' "bw_max" : 156036,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'155680.285714,\n'
' "bw_dev" : 179.364017,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 310822,\n'
' "iops_max" : 312072,\n'
' "iops_mean" : '
'311360.571429,\n'
' "iops_stddev" : '
'358.728034,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.085714,\n'
' "sys_cpu" : 82.542857,\n'
' "ctx" : 12650,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010281,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.110702,\n'
' "10" : 79.515885,\n'
' "20" : 18.645368,\n'
' "50" : 1.666728,\n'
' "100" : 0.042362,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 3132143,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 15922,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 15922,\n'
' "util" : 99.098564\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=141996: Wed '
'Oct 26 04:40:25 2022\n'
' read: IOPS=311k, BW=152MiB/s '
'(159MB/s)(1064MiB/7001msec)\n'
' slat (nsec): min=1292, '
'max=28123, avg=2499.69, '
'stdev=889.90\n'
' clat (nsec): min=70, '
'max=142838, avg=9952.94, '
'stdev=2613.80\n'
' lat (usec): min=5, max=144, '
'avg=12.45, stdev= 2.61\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7392], '
'5.00th=[ 8032], 10.00th=[ 9152], '
'20.00th=[ 9280],\n'
' | 30.00th=[ 9408], '
'40.00th=[ 9408], 50.00th=[ '
'9536], 60.00th=[ 9536],\n'
' | 70.00th=[ 9536], '
'80.00th=[10432], '
'90.00th=[11072], '
'95.00th=[11328],\n'
' | 99.00th=[23168], '
'99.50th=[24960], '
'99.90th=[29824], '
'99.95th=[34560],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=155411, '
'max=156036, per=100.00%, '
'avg=155680.29, stdev=179.36, '
'samples=14\n'
' iops : min=310822, '
'max=312072, avg=311360.57, '
'stdev=358.73, samples=14\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.11%, 10=79.52%, 20=18.65%, '
'50=1.67%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=15.09%, '
'sys=82.54%, ctx=12650, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2178820,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=152MiB/s (159MB/s), '
'152MiB/s-152MiB/s '
'(159MB/s-159MB/s), io=1064MiB '
'(1116MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=3132143/0, '
'merge=0/0, ticks=15922/0, '
'in_queue=15922, util=99.10%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.output'),
'state': {'begin': 1666744936.8076468,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7865753173828125,
'end': 1666744937.594222,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.state')},
'cmd_232': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.output'),
'state': {'begin': 1666744938.592386,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.79380202293396,
'end': 1666744939.386188,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.state')},
'cmd_233': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.output'),
'state': {'begin': 1666744939.3881023,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.045184850692749,
'end': 1666744950.4332871,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.state')},
'cmd_234': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752039,\n'
' "timestamp_ms" : '
'1666752039618,\n'
' "time" : "Wed Oct 26 04:40:39 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1114947584,\n'
' "io_kbytes" : 1088816,\n'
' "bw_bytes" : 159255475,\n'
' "bw" : 155522,\n'
' "iops" : 311045.279246,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2177628,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1282,\n'
' "max" : 115326,\n'
' "mean" : 2502.982368,\n'
' "stddev" : '
'919.655007,\n'
' "N" : 2177629\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 70,\n'
' "max" : 153448,\n'
' "mean" : 9965.891609,\n'
' "stddev" : '
'2636.066476,\n'
' "N" : 2177631,\n'
' "percentile" : {\n'
' "1.000000" : 7456,\n'
' "5.000000" : 8032,\n'
' "10.000000" : 9152,\n'
' "20.000000" : 9280,\n'
' "30.000000" : 9408,\n'
' "40.000000" : 9408,\n'
' "50.000000" : 9536,\n'
' "60.000000" : 9536,\n'
' "70.000000" : 9536,\n'
' "80.000000" : 9920,\n'
' "90.000000" : '
'11072,\n'
' "95.000000" : '
'11200,\n'
' "99.000000" : '
'23168,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'29312,\n'
' "99.950000" : '
'34048,\n'
' "99.990000" : 73216\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5721,\n'
' "max" : 154770,\n'
' "mean" : '
'12468.873560,\n'
' "stddev" : '
'2639.118189,\n'
' "N" : 2177631\n'
' },\n'
' "bw_min" : 155227,\n'
' "bw_max" : 156354,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'155585.285714,\n'
' "bw_dev" : 335.643453,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 310454,\n'
' "iops_max" : 312710,\n'
' "iops_mean" : '
'311170.571429,\n'
' "iops_stddev" : '
'671.748776,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.314286,\n'
' "sys_cpu" : 82.328571,\n'
' "ctx" : 12629,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.015154,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.107319,\n'
' "10" : 79.928757,\n'
' "20" : 18.224600,\n'
' "50" : 1.676457,\n'
' "100" : 0.037380,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 3131615,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 15931,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 15931,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=142034: Wed '
'Oct 26 04:40:39 2022\n'
' read: IOPS=311k, BW=152MiB/s '
'(159MB/s)(1063MiB/7001msec)\n'
' slat (nsec): min=1282, '
'max=115326, avg=2502.98, '
'stdev=919.66\n'
' clat (nsec): min=70, '
'max=153448, avg=9965.89, '
'stdev=2636.07\n'
' lat (usec): min=5, max=154, '
'avg=12.47, stdev= 2.64\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7456], '
'5.00th=[ 8032], 10.00th=[ 9152], '
'20.00th=[ 9280],\n'
' | 30.00th=[ 9408], '
'40.00th=[ 9408], 50.00th=[ '
'9536], 60.00th=[ 9536],\n'
' | 70.00th=[ 9536], '
'80.00th=[ 9920], '
'90.00th=[11072], '
'95.00th=[11200],\n'
' | 99.00th=[23168], '
'99.50th=[24960], '
'99.90th=[29312], '
'99.95th=[34048],\n'
' | 99.99th=[73216]\n'
' bw ( KiB/s): min=155227, '
'max=156354, per=100.00%, '
'avg=155585.29, stdev=335.64, '
'samples=14\n'
' iops : min=310454, '
'max=312710, avg=311170.57, '
'stdev=671.75, samples=14\n'
' lat (nsec) : 100=0.02%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.11%, 10=79.93%, 20=18.22%, '
'50=1.68%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=15.31%, '
'sys=82.33%, ctx=12629, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2177628,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=152MiB/s (159MB/s), '
'152MiB/s-152MiB/s '
'(159MB/s-159MB/s), io=1063MiB '
'(1115MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=3131615/0, '
'merge=0/0, ticks=15931/0, '
'in_queue=15931, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.output'),
'state': {'begin': 1666744950.435849,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7989151477813721,
'end': 1666744951.234764,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.state')},
'cmd_235': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.output'),
'state': {'begin': 1666744953.2707112,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7816410064697266,
'end': 1666744954.0523522,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.state')},
'cmd_236': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.output'),
'state': {'begin': 1666744954.0543783,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.445031881332397,
'end': 1666744965.4994102,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.state')},
'cmd_237': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752054,\n'
' "timestamp_ms" : '
'1666752054612,\n'
' "time" : "Wed Oct 26 04:40:54 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1012919808,\n'
' "io_kbytes" : 989179,\n'
' "bw_bytes" : 144682160,\n'
' "bw" : 141291,\n'
' "iops" : 282581.916869,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1978356,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 26279,\n'
' "mean" : 141.316532,\n'
' "stddev" : 49.171458,\n'
' "N" : 1978356\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 11882,\n'
' "max" : 177744,\n'
' "mean" : '
'13172.391741,\n'
' "stddev" : '
'2083.866987,\n'
' "N" : 1978356,\n'
' "percentile" : {\n'
' "1.000000" : 12224,\n'
' "5.000000" : 12352,\n'
' "10.000000" : '
'12352,\n'
' "20.000000" : '
'12608,\n'
' "30.000000" : '
'12608,\n'
' "40.000000" : '
'12736,\n'
' "50.000000" : '
'12864,\n'
' "60.000000" : '
'12992,\n'
' "70.000000" : '
'13120,\n'
' "80.000000" : '
'13248,\n'
' "90.000000" : '
'13376,\n'
' "95.000000" : '
'14016,\n'
' "99.000000" : '
'22912,\n'
' "99.500000" : '
'23936,\n'
' "99.900000" : '
'30592,\n'
' "99.950000" : '
'46336,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 12003,\n'
' "max" : 177944,\n'
' "mean" : '
'13313.708273,\n'
' "stddev" : '
'2088.192140,\n'
' "N" : 1978356\n'
' },\n'
' "bw_min" : 141168,\n'
' "bw_max" : 141492,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'141330.928571,\n'
' "bw_dev" : 101.098934,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 282336,\n'
' "iops_max" : 282984,\n'
' "iops_mean" : '
'282661.928571,\n'
' "iops_stddev" : '
'201.869137,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 38.900000,\n'
' "sys_cpu" : 61.071429,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 25.000000,\n'
' "2" : 50.000000,\n'
' "4" : 25.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 98.583622,\n'
' "50" : 1.369925,\n'
' "100" : 0.039477,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=142084: Wed '
'Oct 26 04:40:54 2022\n'
' read: IOPS=283k, BW=138MiB/s '
'(145MB/s)(966MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=26279, avg=141.32, '
'stdev=49.17\n'
' clat (usec): min=11, '
'max=177, avg=13.17, stdev= 2.08\n'
' lat (usec): min=12, '
'max=177, avg=13.31, stdev= 2.09\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12224], '
'5.00th=[12352], 10.00th=[12352], '
'20.00th=[12608],\n'
' | 30.00th=[12608], '
'40.00th=[12736], '
'50.00th=[12864], '
'60.00th=[12992],\n'
' | 70.00th=[13120], '
'80.00th=[13248], '
'90.00th=[13376], '
'95.00th=[14016],\n'
' | 99.00th=[22912], '
'99.50th=[23936], '
'99.90th=[30592], '
'99.95th=[46336],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=141168, '
'max=141492, per=100.00%, '
'avg=141330.93, stdev=101.10, '
'samples=14\n'
' iops : min=282336, '
'max=282984, avg=282661.93, '
'stdev=201.87, samples=14\n'
' lat (usec) : 20=98.58%, '
'50=1.37%, 100=0.04%, 250=0.01%\n'
' cpu : usr=38.90%, '
'sys=61.07%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=25.0%, '
'2=50.0%, 4=25.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1978356,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=138MiB/s (145MB/s), '
'138MiB/s-138MiB/s '
'(145MB/s-145MB/s), io=966MiB '
'(1013MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.output'),
'state': {'begin': 1666744965.5021877,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7887923717498779,
'end': 1666744966.29098,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.state')},
'cmd_238': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.output'),
'state': {'begin': 1666744968.3584857,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7827889919281006,
'end': 1666744969.1412747,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.state')},
'cmd_239': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.output'),
'state': {'begin': 1666744969.1431932,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.46291971206665,
'end': 1666744980.606113,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.state')},
'cmd_24': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751038,\n'
' "timestamp_ms" : '
'1666751038438,\n'
' "time" : "Wed Oct 26 04:23:58 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 438721024,\n'
' "io_kbytes" : 428438,\n'
' "bw_bytes" : 62665479,\n'
' "bw" : 61196,\n'
' "iops" : 122393.372375,\n'
' "runtime" : 7001,\n'
' "total_ios" : 856876,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 711,\n'
' "max" : 19336,\n'
' "mean" : 922.608556,\n'
' "stddev" : 65.998960,\n'
' "N" : 856877\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 731,\n'
' "max" : 1292413,\n'
' "mean" : 6966.034914,\n'
' "stddev" : '
'1782.190180,\n'
' "N" : 856876,\n'
' "percentile" : {\n'
' "1.000000" : 6368,\n'
' "5.000000" : 6496,\n'
' "10.000000" : 6560,\n'
' "20.000000" : 6624,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 6752,\n'
' "50.000000" : 6816,\n'
' "60.000000" : 6944,\n'
' "70.000000" : 6944,\n'
' "80.000000" : 7008,\n'
' "90.000000" : 7136,\n'
' "95.000000" : 7264,\n'
' "99.000000" : 12864,\n'
' "99.500000" : 13888,\n'
' "99.900000" : 16512,\n'
' "99.950000" : 16768,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6993,\n'
' "max" : 1293966,\n'
' "mean" : 7888.643483,\n'
' "stddev" : '
'1784.824087,\n'
' "N" : 856876\n'
' },\n'
' "bw_min" : 61040,\n'
' "bw_max" : 61478,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'61222.769231,\n'
' "bw_dev" : 152.265751,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 122080,\n'
' "iops_max" : 122956,\n'
' "iops_mean" : '
'122445.769231,\n'
' "iops_stddev" : '
'304.518241,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 21,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.313175,\n'
' "20" : 1.668036,\n'
' "50" : 0.010000,\n'
' "100" : 0.013771,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=138968: Wed Oct 26 04:23:58 '
'2022\n'
' read: IOPS=122k, BW=59.8MiB/s '
'(62.7MB/s)(418MiB/7001msec)\n'
' slat (nsec): min=711, '
'max=19336, avg=922.61, '
'stdev=66.00\n'
' clat (nsec): min=731, '
'max=1292.4k, avg=6966.03, '
'stdev=1782.19\n'
' lat (usec): min=6, max=1293, '
'avg= 7.89, stdev= 1.78\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6368], 5.00th=[ '
'6496], 10.00th=[ 6560], 20.00th=[ '
'6624],\n'
' | 30.00th=[ 6752], 40.00th=[ '
'6752], 50.00th=[ 6816], 60.00th=[ '
'6944],\n'
' | 70.00th=[ 6944], 80.00th=[ '
'7008], 90.00th=[ 7136], 95.00th=[ '
'7264],\n'
' | 99.00th=[12864], '
'99.50th=[13888], 99.90th=[16512], '
'99.95th=[16768],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=61040, '
'max=61478, per=100.00%, '
'avg=61222.77, stdev=152.27, '
'samples=13\n'
' iops : min=122080, '
'max=122956, avg=122445.77, '
'stdev=304.52, samples=13\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 10=98.31%, '
'20=1.67%, 50=0.01%, 100=0.01%\n'
' lat (msec) : 2=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=21, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=856876,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=59.8MiB/s (62.7MB/s), '
'59.8MiB/s-59.8MiB/s '
'(62.7MB/s-62.7MB/s), io=418MiB '
'(439MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.output'),
'state': {'begin': 1666743949.3106942,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.787369966506958,
'end': 1666743950.0980642,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.state')},
'cmd_240': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752069,\n'
' "timestamp_ms" : '
'1666752069718,\n'
' "time" : "Wed Oct 26 04:41:09 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1013155328,\n'
' "io_kbytes" : 989409,\n'
' "bw_bytes" : 144715801,\n'
' "bw" : 141324,\n'
' "iops" : 282647.621768,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1978816,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 12503,\n'
' "mean" : 142.496506,\n'
' "stddev" : 47.180392,\n'
' "N" : 1978816\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 11873,\n'
' "max" : 117370,\n'
' "mean" : '
'13154.827341,\n'
' "stddev" : '
'1841.054914,\n'
' "N" : 1978816,\n'
' "percentile" : {\n'
' "1.000000" : 12224,\n'
' "5.000000" : 12352,\n'
' "10.000000" : '
'12352,\n'
' "20.000000" : '
'12608,\n'
' "30.000000" : '
'12608,\n'
' "40.000000" : '
'12736,\n'
' "50.000000" : '
'12864,\n'
' "60.000000" : '
'12992,\n'
' "70.000000" : '
'13120,\n'
' "80.000000" : '
'13248,\n'
' "90.000000" : '
'13376,\n'
' "95.000000" : '
'13888,\n'
' "99.000000" : '
'22912,\n'
' "99.500000" : '
'23936,\n'
' "99.900000" : '
'29568,\n'
' "99.950000" : '
'35072,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11993,\n'
' "max" : 117570,\n'
' "mean" : '
'13297.323847,\n'
' "stddev" : '
'1846.116285,\n'
' "N" : 1978816\n'
' },\n'
' "bw_min" : 141129,\n'
' "bw_max" : 141560,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'141360.928571,\n'
' "bw_dev" : 153.096483,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 282256,\n'
' "iops_max" : 283120,\n'
' "iops_mean" : '
'282722.000000,\n'
' "iops_stddev" : '
'306.117122,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 41.242857,\n'
' "sys_cpu" : 58.728571,\n'
' "ctx" : 20,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 25.000000,\n'
' "2" : 50.000000,\n'
' "4" : 25.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 98.730200,\n'
' "50" : 1.228361,\n'
' "100" : 0.041237,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=142133: Wed '
'Oct 26 04:41:09 2022\n'
' read: IOPS=283k, BW=138MiB/s '
'(145MB/s)(966MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=12503, avg=142.50, '
'stdev=47.18\n'
' clat (usec): min=11, '
'max=117, avg=13.15, stdev= 1.84\n'
' lat (usec): min=11, '
'max=117, avg=13.30, stdev= 1.85\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12224], '
'5.00th=[12352], 10.00th=[12352], '
'20.00th=[12608],\n'
' | 30.00th=[12608], '
'40.00th=[12736], '
'50.00th=[12864], '
'60.00th=[12992],\n'
' | 70.00th=[13120], '
'80.00th=[13248], '
'90.00th=[13376], '
'95.00th=[13888],\n'
' | 99.00th=[22912], '
'99.50th=[23936], '
'99.90th=[29568], '
'99.95th=[35072],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=141129, '
'max=141560, per=100.00%, '
'avg=141360.93, stdev=153.10, '
'samples=14\n'
' iops : min=282256, '
'max=283120, avg=282722.00, '
'stdev=306.12, samples=14\n'
' lat (usec) : 20=98.73%, '
'50=1.23%, 100=0.04%, 250=0.01%\n'
' cpu : usr=41.24%, '
'sys=58.73%, ctx=20, majf=0, '
'minf=0\n'
' IO depths : 1=25.0%, '
'2=50.0%, 4=25.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1978816,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=138MiB/s (145MB/s), '
'138MiB/s-138MiB/s '
'(145MB/s-145MB/s), io=966MiB '
'(1013MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.output'),
'state': {'begin': 1666744980.6090484,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7900142669677734,
'end': 1666744981.3990626,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.state')},
'cmd_241': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.output'),
'state': {'begin': 1666744983.4507341,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7813313007354736,
'end': 1666744984.2320654,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.state')},
'cmd_242': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.output'),
'state': {'begin': 1666744984.234375,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.442209720611572,
'end': 1666744995.6765847,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.state')},
'cmd_243': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752084,\n'
' "timestamp_ms" : '
'1666752084789,\n'
' "time" : "Wed Oct 26 04:41:24 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1017103872,\n'
' "io_kbytes" : 993265,\n'
' "bw_bytes" : 145279798,\n'
' "bw" : 141874,\n'
' "iops" : 283749.178689,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1986528,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 13124,\n'
' "mean" : 141.327949,\n'
' "stddev" : 46.350036,\n'
' "N" : 1986528\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 11853,\n'
' "max" : 118873,\n'
' "mean" : '
'13131.754659,\n'
' "stddev" : '
'1847.869899,\n'
' "N" : 1986528,\n'
' "percentile" : {\n'
' "1.000000" : 12224,\n'
' "5.000000" : 12352,\n'
' "10.000000" : '
'12352,\n'
' "20.000000" : '
'12480,\n'
' "30.000000" : '
'12608,\n'
' "40.000000" : '
'12736,\n'
' "50.000000" : '
'12864,\n'
' "60.000000" : '
'12992,\n'
' "70.000000" : '
'12992,\n'
' "80.000000" : '
'13120,\n'
' "90.000000" : '
'13376,\n'
' "95.000000" : '
'13888,\n'
' "99.000000" : '
'22912,\n'
' "99.500000" : '
'23936,\n'
' "99.900000" : '
'29568,\n'
' "99.950000" : '
'35072,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 11963,\n'
' "max" : 119083,\n'
' "mean" : '
'13273.082608,\n'
' "stddev" : '
'1853.093830,\n'
' "N" : 1986528\n'
' },\n'
' "bw_min" : 141696,\n'
' "bw_max" : 142276,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'141915.000000,\n'
' "bw_dev" : 201.676817,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 283392,\n'
' "iops_max" : 284552,\n'
' "iops_mean" : '
'283829.928571,\n'
' "iops_stddev" : '
'403.275906,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 38.557143,\n'
' "sys_cpu" : 61.414286,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 25.000000,\n'
' "2" : 50.000000,\n'
' "4" : 25.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 98.732412,\n'
' "50" : 1.227670,\n'
' "100" : 0.039718,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=142182: Wed '
'Oct 26 04:41:24 2022\n'
' read: IOPS=284k, BW=139MiB/s '
'(145MB/s)(970MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=13124, avg=141.33, '
'stdev=46.35\n'
' clat (usec): min=11, '
'max=118, avg=13.13, stdev= 1.85\n'
' lat (usec): min=11, '
'max=119, avg=13.27, stdev= 1.85\n'
' clat percentiles (nsec):\n'
' | 1.00th=[12224], '
'5.00th=[12352], 10.00th=[12352], '
'20.00th=[12480],\n'
' | 30.00th=[12608], '
'40.00th=[12736], '
'50.00th=[12864], '
'60.00th=[12992],\n'
' | 70.00th=[12992], '
'80.00th=[13120], '
'90.00th=[13376], '
'95.00th=[13888],\n'
' | 99.00th=[22912], '
'99.50th=[23936], '
'99.90th=[29568], '
'99.95th=[35072],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=141696, '
'max=142276, per=100.00%, '
'avg=141915.00, stdev=201.68, '
'samples=14\n'
' iops : min=283392, '
'max=284552, avg=283829.93, '
'stdev=403.28, samples=14\n'
' lat (usec) : 20=98.73%, '
'50=1.23%, 100=0.04%, 250=0.01%\n'
' cpu : usr=38.56%, '
'sys=61.41%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=25.0%, '
'2=50.0%, 4=25.0%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1986528,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=139MiB/s (145MB/s), '
'139MiB/s-139MiB/s '
'(145MB/s-145MB/s), io=970MiB '
'(1017MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.output'),
'state': {'begin': 1666744995.6792803,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7905035018920898,
'end': 1666744996.4697838,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.state')},
'cmd_244': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.output'),
'state': {'begin': 1666744998.5398793,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7882885932922363,
'end': 1666744999.328168,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.state')},
'cmd_245': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.output'),
'state': {'begin': 1666744999.3303547,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.473461866378784,
'end': 1666745010.8038166,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.state')},
'cmd_246': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752099,\n'
' "timestamp_ms" : '
'1666752099916,\n'
' "time" : "Wed Oct 26 04:41:39 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1077817344,\n'
' "io_kbytes" : 1052556,\n'
' "bw_bytes" : 153951913,\n'
' "bw" : 150343,\n'
' "iops" : 300686.759034,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2105108,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1532,\n'
' "max" : 20038,\n'
' "mean" : 2804.261997,\n'
' "stddev" : '
'1140.383516,\n'
' "N" : 2105109\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 190,\n'
' "max" : 124213,\n'
' "mean" : '
'10260.727332,\n'
' "stddev" : '
'2255.624306,\n'
' "N" : 2105111,\n'
' "percentile" : {\n'
' "1.000000" : 8512,\n'
' "5.000000" : 8640,\n'
' "10.000000" : 8896,\n'
' "20.000000" : 9792,\n'
' "30.000000" : 9792,\n'
' "40.000000" : 9920,\n'
' "50.000000" : 9920,\n'
' "60.000000" : 9920,\n'
' "70.000000" : '
'10048,\n'
' "80.000000" : '
'10176,\n'
' "90.000000" : '
'11200,\n'
' "95.000000" : '
'11456,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'23424,\n'
' "99.900000" : '
'29824,\n'
' "99.950000" : '
'38144,\n'
' "99.990000" : 68096\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5651,\n'
' "max" : 125786,\n'
' "mean" : '
'13064.989316,\n'
' "stddev" : '
'2336.079254,\n'
' "N" : 2105111\n'
' },\n'
' "bw_min" : 150030,\n'
' "bw_max" : 150660,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'150378.142857,\n'
' "bw_dev" : 229.647508,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 300060,\n'
' "iops_max" : 301320,\n'
' "iops_mean" : '
'300756.285714,\n'
' "iops_stddev" : '
'459.295016,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.271429,\n'
' "sys_cpu" : 67.700000,\n'
' "ctx" : 21,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.013824,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.021139,\n'
' "10" : 66.454738,\n'
' "20" : 32.398053,\n'
' "50" : 1.065931,\n'
' "100" : 0.044938,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142231: Wed '
'Oct 26 04:41:39 2022\n'
' read: IOPS=301k, BW=147MiB/s '
'(154MB/s)(1028MiB/7001msec)\n'
' slat (nsec): min=1532, '
'max=20038, avg=2804.26, '
'stdev=1140.38\n'
' clat (nsec): min=190, '
'max=124213, avg=10260.73, '
'stdev=2255.62\n'
' lat (usec): min=5, max=125, '
'avg=13.06, stdev= 2.34\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8512], '
'5.00th=[ 8640], 10.00th=[ 8896], '
'20.00th=[ 9792],\n'
' | 30.00th=[ 9792], '
'40.00th=[ 9920], 50.00th=[ '
'9920], 60.00th=[ 9920],\n'
' | 70.00th=[10048], '
'80.00th=[10176], '
'90.00th=[11200], '
'95.00th=[11456],\n'
' | 99.00th=[20864], '
'99.50th=[23424], '
'99.90th=[29824], '
'99.95th=[38144],\n'
' | 99.99th=[68096]\n'
' bw ( KiB/s): min=150030, '
'max=150660, per=100.00%, '
'avg=150378.14, stdev=229.65, '
'samples=14\n'
' iops : min=300060, '
'max=301320, avg=300756.29, '
'stdev=459.30, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.02%, '
'10=66.45%, 20=32.40%, 50=1.07%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=32.27%, '
'sys=67.70%, ctx=21, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2105108,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=147MiB/s (154MB/s), '
'147MiB/s-147MiB/s '
'(154MB/s-154MB/s), io=1028MiB '
'(1078MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.output'),
'state': {'begin': 1666745010.8066304,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.9443705081939697,
'end': 1666745011.751001,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.state')},
'cmd_247': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.output'),
'state': {'begin': 1666745013.8148746,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7813012599945068,
'end': 1666745014.596176,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.state')},
'cmd_248': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.output'),
'state': {'begin': 1666745014.5983183,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.47979736328125,
'end': 1666745026.0781157,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.state')},
'cmd_249': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752115,\n'
' "timestamp_ms" : '
'1666752115189,\n'
' "time" : "Wed Oct 26 04:41:55 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1060443136,\n'
' "io_kbytes" : 1035589,\n'
' "bw_bytes" : 151470237,\n'
' "bw" : 147920,\n'
' "iops" : 295839.737180,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2071174,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1542,\n'
' "max" : 28865,\n'
' "mean" : 2851.573909,\n'
' "stddev" : '
'1251.286691,\n'
' "N" : 2071175\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 190,\n'
' "max" : 120766,\n'
' "mean" : '
'10416.253545,\n'
' "stddev" : '
'2328.765258,\n'
' "N" : 2071177,\n'
' "percentile" : {\n'
' "1.000000" : 8640,\n'
' "5.000000" : 8768,\n'
' "10.000000" : 8768,\n'
' "20.000000" : 8896,\n'
' "30.000000" : 9024,\n'
' "40.000000" : 9920,\n'
' "50.000000" : '
'10048,\n'
' "60.000000" : '
'10176,\n'
' "70.000000" : '
'11072,\n'
' "80.000000" : '
'11328,\n'
' "90.000000" : '
'11456,\n'
' "95.000000" : '
'11584,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'23424,\n'
' "99.900000" : '
'29824,\n'
' "99.950000" : '
'34048,\n'
' "99.990000" : 66048\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5650,\n'
' "max" : 122359,\n'
' "mean" : '
'13267.826877,\n'
' "stddev" : '
'2220.362076,\n'
' "N" : 2071177\n'
' },\n'
' "bw_min" : 147590,\n'
' "bw_max" : 148421,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'147953.571429,\n'
' "bw_dev" : 238.447451,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 295180,\n'
' "iops_max" : 296842,\n'
' "iops_mean" : '
'295907.071429,\n'
' "iops_stddev" : '
'476.984513,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.214286,\n'
' "sys_cpu" : 69.757143,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.016512,\n'
' "10" : 42.578412,\n'
' "20" : 56.246361,\n'
' "50" : 1.119945,\n'
' "100" : 0.037129,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142280: Wed '
'Oct 26 04:41:55 2022\n'
' read: IOPS=296k, BW=144MiB/s '
'(151MB/s)(1011MiB/7001msec)\n'
' slat (nsec): min=1542, '
'max=28865, avg=2851.57, '
'stdev=1251.29\n'
' clat (nsec): min=190, '
'max=120766, avg=10416.25, '
'stdev=2328.77\n'
' lat (usec): min=5, max=122, '
'avg=13.27, stdev= 2.22\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8640], '
'5.00th=[ 8768], 10.00th=[ 8768], '
'20.00th=[ 8896],\n'
' | 30.00th=[ 9024], '
'40.00th=[ 9920], '
'50.00th=[10048], '
'60.00th=[10176],\n'
' | 70.00th=[11072], '
'80.00th=[11328], '
'90.00th=[11456], '
'95.00th=[11584],\n'
' | 99.00th=[20864], '
'99.50th=[23424], '
'99.90th=[29824], '
'99.95th=[34048],\n'
' | 99.99th=[66048]\n'
' bw ( KiB/s): min=147590, '
'max=148421, per=100.00%, '
'avg=147953.57, stdev=238.45, '
'samples=14\n'
' iops : min=295180, '
'max=296842, avg=295907.07, '
'stdev=476.98, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.02%, '
'10=42.58%, 20=56.25%, 50=1.12%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=30.21%, '
'sys=69.76%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2071174,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=144MiB/s (151MB/s), '
'144MiB/s-144MiB/s '
'(151MB/s-151MB/s), io=1011MiB '
'(1060MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.output'),
'state': {'begin': 1666745026.081021,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7926065921783447,
'end': 1666745026.8736277,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.state')},
'cmd_25': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.output'),
'state': {'begin': 1666743952.1315753,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7893722057342529,
'end': 1666743952.9209476,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.state')},
'cmd_250': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.output'),
'state': {'begin': 1666745028.9053721,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7826261520385742,
'end': 1666745029.6879983,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.state')},
'cmd_251': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.output'),
'state': {'begin': 1666745029.6903412,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.451205968856812,
'end': 1666745041.1415472,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.state')},
'cmd_252': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752130,\n'
' "timestamp_ms" : '
'1666752130258,\n'
' "time" : "Wed Oct 26 04:42:10 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1063592960,\n'
' "io_kbytes" : 1038665,\n'
' "bw_bytes" : 151920148,\n'
' "bw" : 148359,\n'
' "iops" : 296718.468790,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2077326,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1533,\n'
' "max" : 57908,\n'
' "mean" : 2828.639420,\n'
' "stddev" : '
'1225.631029,\n'
' "N" : 2077327\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 210,\n'
' "max" : 121017,\n'
' "mean" : '
'10398.118086,\n'
' "stddev" : '
'2330.860370,\n'
' "N" : 2077329,\n'
' "percentile" : {\n'
' "1.000000" : 8640,\n'
' "5.000000" : 8768,\n'
' "10.000000" : 8768,\n'
' "20.000000" : 8896,\n'
' "30.000000" : 9920,\n'
' "40.000000" : 9920,\n'
' "50.000000" : '
'10048,\n'
' "60.000000" : '
'10176,\n'
' "70.000000" : '
'10944,\n'
' "80.000000" : '
'11200,\n'
' "90.000000" : '
'11328,\n'
' "95.000000" : '
'11584,\n'
' "99.000000" : '
'20864,\n'
' "99.500000" : '
'23424,\n'
' "99.900000" : '
'30080,\n'
' "99.950000" : '
'35072,\n'
' "99.990000" : 67072\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5670,\n'
' "max" : 122710,\n'
' "mean" : '
'13226.758125,\n'
' "stddev" : '
'2280.945948,\n'
' "N" : 2077329\n'
' },\n'
' "bw_min" : 147572,\n'
' "bw_max" : 148824,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'148401.285714,\n'
' "bw_dev" : 298.527522,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 295144,\n'
' "iops_max" : 297649,\n'
' "iops_mean" : '
'296802.642857,\n'
' "iops_stddev" : '
'597.164017,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 32.328571,\n'
' "sys_cpu" : 67.642857,\n'
' "ctx" : 29,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.023925,\n'
' "10" : 44.046240,\n'
' "20" : 54.765116,\n'
' "50" : 1.121779,\n'
' "100" : 0.039426,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142330: Wed '
'Oct 26 04:42:10 2022\n'
' read: IOPS=297k, BW=145MiB/s '
'(152MB/s)(1014MiB/7001msec)\n'
' slat (nsec): min=1533, '
'max=57908, avg=2828.64, '
'stdev=1225.63\n'
' clat (nsec): min=210, '
'max=121017, avg=10398.12, '
'stdev=2330.86\n'
' lat (usec): min=5, max=122, '
'avg=13.23, stdev= 2.28\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8640], '
'5.00th=[ 8768], 10.00th=[ 8768], '
'20.00th=[ 8896],\n'
' | 30.00th=[ 9920], '
'40.00th=[ 9920], '
'50.00th=[10048], '
'60.00th=[10176],\n'
' | 70.00th=[10944], '
'80.00th=[11200], '
'90.00th=[11328], '
'95.00th=[11584],\n'
' | 99.00th=[20864], '
'99.50th=[23424], '
'99.90th=[30080], '
'99.95th=[35072],\n'
' | 99.99th=[67072]\n'
' bw ( KiB/s): min=147572, '
'max=148824, per=100.00%, '
'avg=148401.29, stdev=298.53, '
'samples=14\n'
' iops : min=295144, '
'max=297649, avg=296802.64, '
'stdev=597.16, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.02%, '
'10=44.05%, 20=54.77%, 50=1.12%, '
'100=0.04%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=32.33%, '
'sys=67.64%, ctx=29, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2077326,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=145MiB/s (152MB/s), '
'145MiB/s-145MiB/s '
'(152MB/s-152MB/s), io=1014MiB '
'(1064MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.output'),
'state': {'begin': 1666745041.1447248,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7829673290252686,
'end': 1666745041.9276922,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.state')},
'cmd_253': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.output'),
'state': {'begin': 1666745042.9159136,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7921304702758789,
'end': 1666745043.708044,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.state')},
'cmd_254': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.output'),
'state': {'begin': 1666745043.71043,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.055681467056274,
'end': 1666745054.7661114,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.state')},
'cmd_255': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752143,\n'
' "timestamp_ms" : '
'1666752143949,\n'
' "time" : "Wed Oct 26 04:42:23 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1086656512,\n'
' "io_kbytes" : 1061188,\n'
' "bw_bytes" : 155214471,\n'
' "bw" : 151576,\n'
' "iops" : 303152.692473,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2122372,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1372,\n'
' "max" : 112582,\n'
' "mean" : 2810.550236,\n'
' "stddev" : '
'1247.731970,\n'
' "N" : 2122373\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 110,\n'
' "max" : 125957,\n'
' "mean" : '
'10142.478753,\n'
' "stddev" : '
'2569.151273,\n'
' "N" : 2122375,\n'
' "percentile" : {\n'
' "1.000000" : 6880,\n'
' "5.000000" : 7968,\n'
' "10.000000" : 8256,\n'
' "20.000000" : 9536,\n'
' "30.000000" : 9664,\n'
' "40.000000" : 9664,\n'
' "50.000000" : 9792,\n'
' "60.000000" : 9792,\n'
' "70.000000" : 9792,\n'
' "80.000000" : '
'10048,\n'
' "90.000000" : '
'11456,\n'
' "95.000000" : '
'11968,\n'
' "99.000000" : '
'23424,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'29824,\n'
' "99.950000" : '
'34048,\n'
' "99.990000" : 70144\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5921,\n'
' "max" : 127369,\n'
' "mean" : '
'12953.029064,\n'
' "stddev" : '
'2798.630057,\n'
' "N" : 2122375\n'
' },\n'
' "bw_min" : 151380,\n'
' "bw_max" : 151845,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'151616.571429,\n'
' "bw_dev" : 148.988236,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 302760,\n'
' "iops_max" : 303691,\n'
' "iops_mean" : '
'303233.214286,\n'
' "iops_stddev" : '
'298.105861,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 18.328571,\n'
' "sys_cpu" : 79.457143,\n'
' "ctx" : 12395,\n'
' "majf" : 0,\n'
' "minf" : 1,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.024925,\n'
' "4" : 0.110160,\n'
' "10" : 80.312170,\n'
' "20" : 17.752307,\n'
' "50" : 1.758127,\n'
' "100" : 0.039578,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142369: Wed '
'Oct 26 04:42:23 2022\n'
' read: IOPS=303k, BW=148MiB/s '
'(155MB/s)(1036MiB/7001msec)\n'
' slat (nsec): min=1372, '
'max=112582, avg=2810.55, '
'stdev=1247.73\n'
' clat (nsec): min=110, '
'max=125957, avg=10142.48, '
'stdev=2569.15\n'
' lat (usec): min=5, max=127, '
'avg=12.95, stdev= 2.80\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6880], '
'5.00th=[ 7968], 10.00th=[ 8256], '
'20.00th=[ 9536],\n'
' | 30.00th=[ 9664], '
'40.00th=[ 9664], 50.00th=[ '
'9792], 60.00th=[ 9792],\n'
' | 70.00th=[ 9792], '
'80.00th=[10048], '
'90.00th=[11456], '
'95.00th=[11968],\n'
' | 99.00th=[23424], '
'99.50th=[24960], '
'99.90th=[29824], '
'99.95th=[34048],\n'
' | 99.99th=[70144]\n'
' bw ( KiB/s): min=151380, '
'max=151845, per=100.00%, '
'avg=151616.57, stdev=148.99, '
'samples=14\n'
' iops : min=302760, '
'max=303691, avg=303233.21, '
'stdev=298.11, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.02%, '
'4=0.11%, 10=80.31%, 20=17.75%, '
'50=1.76%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=18.33%, '
'sys=79.46%, ctx=12395, majf=0, '
'minf=1\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2122372,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=148MiB/s (155MB/s), '
'148MiB/s-148MiB/s '
'(155MB/s-155MB/s), io=1036MiB '
'(1087MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.output'),
'state': {'begin': 1666745054.7728965,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8778285980224609,
'end': 1666745055.6507251,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.state')},
'cmd_256': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.output'),
'state': {'begin': 1666745056.7810028,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8336989879608154,
'end': 1666745057.6147017,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.state')},
'cmd_257': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.output'),
'state': {'begin': 1666745057.6171892,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.133683919906616,
'end': 1666745068.750873,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.state')},
'cmd_258': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752157,\n'
' "timestamp_ms" : '
'1666752157934,\n'
' "time" : "Wed Oct 26 04:42:37 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1083688448,\n'
' "io_kbytes" : 1058289,\n'
' "bw_bytes" : 154790522,\n'
' "bw" : 151162,\n'
' "iops" : 302324.667905,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2116575,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1362,\n'
' "max" : 21871,\n'
' "mean" : 2810.793388,\n'
' "stddev" : '
'1238.113474,\n'
' "N" : 2116576\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 110,\n'
' "max" : 126127,\n'
' "mean" : '
'10178.077711,\n'
' "stddev" : '
'2575.128921,\n'
' "N" : 2116578,\n'
' "percentile" : {\n'
' "1.000000" : 6880,\n'
' "5.000000" : 7968,\n'
' "10.000000" : 8512,\n'
' "20.000000" : 9536,\n'
' "30.000000" : 9664,\n'
' "40.000000" : 9664,\n'
' "50.000000" : 9792,\n'
' "60.000000" : 9792,\n'
' "70.000000" : 9920,\n'
' "80.000000" : '
'10048,\n'
' "90.000000" : '
'11456,\n'
' "95.000000" : '
'11968,\n'
' "99.000000" : '
'23424,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'30336,\n'
' "99.950000" : '
'34560,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5841,\n'
' "max" : 127520,\n'
' "mean" : '
'12988.870477,\n'
' "stddev" : '
'2788.918764,\n'
' "N" : 2116578\n'
' },\n'
' "bw_min" : 150921,\n'
' "bw_max" : 151477,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'151202.642857,\n'
' "bw_dev" : 172.050801,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 301842,\n'
' "iops_max" : 302954,\n'
' "iops_mean" : '
'302405.500000,\n'
' "iops_stddev" : '
'344.057408,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.585714,\n'
' "sys_cpu" : 80.171429,\n'
' "ctx" : 12447,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.019702,\n'
' "4" : 0.111123,\n'
' "10" : 79.619905,\n'
' "20" : 18.496203,\n'
' "50" : 1.703979,\n'
' "100" : 0.044884,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142408: Wed '
'Oct 26 04:42:37 2022\n'
' read: IOPS=302k, BW=148MiB/s '
'(155MB/s)(1033MiB/7001msec)\n'
' slat (nsec): min=1362, '
'max=21871, avg=2810.79, '
'stdev=1238.11\n'
' clat (nsec): min=110, '
'max=126127, avg=10178.08, '
'stdev=2575.13\n'
' lat (usec): min=5, max=127, '
'avg=12.99, stdev= 2.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6880], '
'5.00th=[ 7968], 10.00th=[ 8512], '
'20.00th=[ 9536],\n'
' | 30.00th=[ 9664], '
'40.00th=[ 9664], 50.00th=[ '
'9792], 60.00th=[ 9792],\n'
' | 70.00th=[ 9920], '
'80.00th=[10048], '
'90.00th=[11456], '
'95.00th=[11968],\n'
' | 99.00th=[23424], '
'99.50th=[24960], '
'99.90th=[30336], '
'99.95th=[34560],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=150921, '
'max=151477, per=100.00%, '
'avg=151202.64, stdev=172.05, '
'samples=14\n'
' iops : min=301842, '
'max=302954, avg=302405.50, '
'stdev=344.06, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.02%, '
'4=0.11%, 10=79.62%, 20=18.50%, '
'50=1.70%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=17.59%, '
'sys=80.17%, ctx=12447, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2116575,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=148MiB/s (155MB/s), '
'148MiB/s-148MiB/s '
'(155MB/s-155MB/s), io=1033MiB '
'(1084MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.output'),
'state': {'begin': 1666745068.7541783,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7893593311309814,
'end': 1666745069.5435376,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.state')},
'cmd_259': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.output'),
'state': {'begin': 1666745070.5367262,
'cmd': 'rm '
'/tmp/fio-output_512_4_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.788844108581543,
'end': 1666745071.3255703,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.state')},
'cmd_26': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.output'),
'state': {'begin': 1666743952.9232578,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="0" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.471574783325195,
'end': 1666743964.3948326,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.state')},
'cmd_260': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.output'),
'state': {'begin': 1666745071.327926,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_io_uring-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.057929515838623,
'end': 1666745082.3858554,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.state')},
'cmd_261': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752171,\n'
' "timestamp_ms" : '
'1666752171569,\n'
' "time" : "Wed Oct 26 04:42:51 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1086826496,\n'
' "io_kbytes" : 1061354,\n'
' "bw_bytes" : 155238751,\n'
' "bw" : 151600,\n'
' "iops" : 303200.114269,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2122704,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1362,\n'
' "max" : 69741,\n'
' "mean" : 2801.643828,\n'
' "stddev" : '
'1238.790601,\n'
' "N" : 2122705\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 110,\n'
' "max" : 124213,\n'
' "mean" : '
'10150.495425,\n'
' "stddev" : '
'2562.103546,\n'
' "N" : 2122707,\n'
' "percentile" : {\n'
' "1.000000" : 6880,\n'
' "5.000000" : 7968,\n'
' "10.000000" : 9408,\n'
' "20.000000" : 9536,\n'
' "30.000000" : 9664,\n'
' "40.000000" : 9664,\n'
' "50.000000" : 9792,\n'
' "60.000000" : 9792,\n'
' "70.000000" : 9920,\n'
' "80.000000" : 9920,\n'
' "90.000000" : '
'11328,\n'
' "95.000000" : '
'11840,\n'
' "99.000000" : '
'23424,\n'
' "99.500000" : '
'24960,\n'
' "99.900000" : '
'30336,\n'
' "99.950000" : '
'34560,\n'
' "99.990000" : 70144\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5861,\n'
' "max" : 125596,\n'
' "mean" : '
'12952.138615,\n'
' "stddev" : '
'2763.745120,\n'
' "N" : 2122707\n'
' },\n'
' "bw_min" : 151397,\n'
' "bw_max" : 151887,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'151658.000000,\n'
' "bw_dev" : 143.826864,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 302794,\n'
' "iops_max" : 303775,\n'
' "iops_mean" : '
'303316.076923,\n'
' "iops_stddev" : '
'287.786513,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.157143,\n'
' "sys_cpu" : 78.600000,\n'
' "ctx" : 12468,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.049371,\n'
' "4" : 0.088990,\n'
' "10" : 81.786580,\n'
' "20" : 16.352115,\n'
' "50" : 1.670982,\n'
' "100" : 0.044519,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142447: Wed '
'Oct 26 04:42:51 2022\n'
' read: IOPS=303k, BW=148MiB/s '
'(155MB/s)(1036MiB/7001msec)\n'
' slat (nsec): min=1362, '
'max=69741, avg=2801.64, '
'stdev=1238.79\n'
' clat (nsec): min=110, '
'max=124213, avg=10150.50, '
'stdev=2562.10\n'
' lat (usec): min=5, max=125, '
'avg=12.95, stdev= 2.76\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6880], '
'5.00th=[ 7968], 10.00th=[ 9408], '
'20.00th=[ 9536],\n'
' | 30.00th=[ 9664], '
'40.00th=[ 9664], 50.00th=[ '
'9792], 60.00th=[ 9792],\n'
' | 70.00th=[ 9920], '
'80.00th=[ 9920], '
'90.00th=[11328], '
'95.00th=[11840],\n'
' | 99.00th=[23424], '
'99.50th=[24960], '
'99.90th=[30336], '
'99.95th=[34560],\n'
' | 99.99th=[70144]\n'
' bw ( KiB/s): min=151397, '
'max=151887, per=100.00%, '
'avg=151658.00, stdev=143.83, '
'samples=13\n'
' iops : min=302794, '
'max=303775, avg=303316.08, '
'stdev=287.79, samples=13\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.05%, '
'4=0.09%, 10=81.79%, 20=16.35%, '
'50=1.67%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=19.16%, '
'sys=78.60%, ctx=12468, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2122704,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=148MiB/s (155MB/s), '
'148MiB/s-148MiB/s '
'(155MB/s-155MB/s), io=1036MiB '
'(1087MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.output'),
'state': {'begin': 1666745082.3891363,
'cmd': 'cat '
'/tmp/fio-output_512_4_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8009669780731201,
'end': 1666745083.1901033,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.state')},
'cmd_262': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.output'),
'state': {'begin': 1666745084.1825252,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7820878028869629,
'end': 1666745084.964613,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.state')},
'cmd_263': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.output'),
'state': {'begin': 1666745084.9666383,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.072291612625122,
'end': 1666745096.03893,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.state')},
'cmd_264': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752185,\n'
' "timestamp_ms" : '
'1666752185221,\n'
' "time" : "Wed Oct 26 04:43:05 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1041431040,\n'
' "io_kbytes" : 1017022,\n'
' "bw_bytes" : 148754612,\n'
' "bw" : 145268,\n'
' "iops" : 290535.780603,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2034041,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1403,\n'
' "max" : 42539,\n'
' "mean" : 2718.299399,\n'
' "stddev" : '
'1303.395648,\n'
' "N" : 2034042\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4479,\n'
' "max" : 124985,\n'
' "mean" : '
'10812.651804,\n'
' "stddev" : '
'2728.741015,\n'
' "N" : 2034044,\n'
' "percentile" : {\n'
' "1.000000" : 9024,\n'
' "5.000000" : 9024,\n'
' "10.000000" : 9024,\n'
' "20.000000" : 9152,\n'
' "30.000000" : 9280,\n'
' "40.000000" : '
'10304,\n'
' "50.000000" : '
'10432,\n'
' "60.000000" : '
'10560,\n'
' "70.000000" : '
'11584,\n'
' "80.000000" : '
'11712,\n'
' "90.000000" : '
'11840,\n'
' "95.000000" : '
'12096,\n'
' "99.000000" : '
'23680,\n'
' "99.500000" : '
'28800,\n'
' "99.900000" : '
'35072,\n'
' "99.950000" : '
'40704,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8055,\n'
' "max" : 129283,\n'
' "mean" : '
'13530.950614,\n'
' "stddev" : '
'2627.157009,\n'
' "N" : 2034044\n'
' },\n'
' "bw_min" : 144981,\n'
' "bw_max" : 145561,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'145329.000000,\n'
' "bw_dev" : 165.697593,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 289962,\n'
' "iops_max" : 291122,\n'
' "iops_mean" : '
'290658.071429,\n'
' "iops_stddev" : '
'331.438465,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 34.800000,\n'
' "sys_cpu" : 63.071429,\n'
' "ctx" : 12160,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 32.978195,\n'
' "20" : 65.542287,\n'
' "50" : 1.435320,\n'
' "100" : 0.043804,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2916402,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 15253,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 15253,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=142487: Wed '
'Oct 26 04:43:05 2022\n'
' read: IOPS=291k, BW=142MiB/s '
'(149MB/s)(993MiB/7001msec)\n'
' slat (nsec): min=1403, '
'max=42539, avg=2718.30, '
'stdev=1303.40\n'
' clat (usec): min=4, max=124, '
'avg=10.81, stdev= 2.73\n'
' lat (usec): min=8, max=129, '
'avg=13.53, stdev= 2.63\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9024], '
'5.00th=[ 9024], 10.00th=[ 9024], '
'20.00th=[ 9152],\n'
' | 30.00th=[ 9280], '
'40.00th=[10304], '
'50.00th=[10432], '
'60.00th=[10560],\n'
' | 70.00th=[11584], '
'80.00th=[11712], '
'90.00th=[11840], '
'95.00th=[12096],\n'
' | 99.00th=[23680], '
'99.50th=[28800], '
'99.90th=[35072], '
'99.95th=[40704],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=144981, '
'max=145561, per=100.00%, '
'avg=145329.00, stdev=165.70, '
'samples=14\n'
' iops : min=289962, '
'max=291122, avg=290658.07, '
'stdev=331.44, samples=14\n'
' lat (usec) : 10=32.98%, '
'20=65.54%, 50=1.44%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=34.80%, '
'sys=63.07%, ctx=12160, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2034041,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=142MiB/s (149MB/s), '
'142MiB/s-142MiB/s '
'(149MB/s-149MB/s), io=993MiB '
'(1041MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2916402/0, '
'merge=0/0, ticks=15253/0, '
'in_queue=15253, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.output'),
'state': {'begin': 1666745096.0416799,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7924807071685791,
'end': 1666745096.8341606,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.state')},
'cmd_265': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.output'),
'state': {'begin': 1666745097.8319213,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7851710319519043,
'end': 1666745098.6170924,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.state')},
'cmd_266': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.output'),
'state': {'begin': 1666745098.6191463,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.066049098968506,
'end': 1666745109.6851954,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.state')},
'cmd_267': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752198,\n'
' "timestamp_ms" : '
'1666752198867,\n'
' "time" : "Wed Oct 26 04:43:18 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1038159872,\n'
' "io_kbytes" : 1013828,\n'
' "bw_bytes" : 148287369,\n'
' "bw" : 144811,\n'
' "iops" : 289623.196686,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2027652,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 25027,\n'
' "mean" : 2728.420388,\n'
' "stddev" : '
'1317.432382,\n'
' "N" : 2027653\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4689,\n'
' "max" : 93515,\n'
' "mean" : '
'10845.511402,\n'
' "stddev" : '
'2729.393787,\n'
' "N" : 2027655,\n'
' "percentile" : {\n'
' "1.000000" : 9024,\n'
' "5.000000" : 9024,\n'
' "10.000000" : 9152,\n'
' "20.000000" : 9152,\n'
' "30.000000" : 9280,\n'
' "40.000000" : '
'10304,\n'
' "50.000000" : '
'10432,\n'
' "60.000000" : '
'10816,\n'
' "70.000000" : '
'11584,\n'
' "80.000000" : '
'11712,\n'
' "90.000000" : '
'11840,\n'
' "95.000000" : '
'11968,\n'
' "99.000000" : '
'23424,\n'
' "99.500000" : '
'29056,\n'
' "99.900000" : '
'35072,\n'
' "99.950000" : '
'40192,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8115,\n'
' "max" : 98064,\n'
' "mean" : '
'13573.932355,\n'
' "stddev" : '
'2591.494403,\n'
' "N" : 2027655\n'
' },\n'
' "bw_min" : 144516,\n'
' "bw_max" : 145113,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'144877.230769,\n'
' "bw_dev" : 161.265906,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 289032,\n'
' "iops_max" : 290226,\n'
' "iops_mean" : '
'289754.692308,\n'
' "iops_stddev" : '
'322.712820,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 34.757143,\n'
' "sys_cpu" : 63.128571,\n'
' "ctx" : 12125,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 36.527964,\n'
' "20" : 61.978979,\n'
' "50" : 1.450989,\n'
' "100" : 0.042216,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2908270,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 15201,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 15201,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=142526: Wed '
'Oct 26 04:43:18 2022\n'
' read: IOPS=290k, BW=141MiB/s '
'(148MB/s)(990MiB/7001msec)\n'
' slat (nsec): min=1402, '
'max=25027, avg=2728.42, '
'stdev=1317.43\n'
' clat (nsec): min=4689, '
'max=93515, avg=10845.51, '
'stdev=2729.39\n'
' lat (nsec): min=8115, '
'max=98064, avg=13573.93, '
'stdev=2591.49\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9024], '
'5.00th=[ 9024], 10.00th=[ 9152], '
'20.00th=[ 9152],\n'
' | 30.00th=[ 9280], '
'40.00th=[10304], '
'50.00th=[10432], '
'60.00th=[10816],\n'
' | 70.00th=[11584], '
'80.00th=[11712], '
'90.00th=[11840], '
'95.00th=[11968],\n'
' | 99.00th=[23424], '
'99.50th=[29056], '
'99.90th=[35072], '
'99.95th=[40192],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=144516, '
'max=145113, per=100.00%, '
'avg=144877.23, stdev=161.27, '
'samples=13\n'
' iops : min=289032, '
'max=290226, avg=289754.69, '
'stdev=322.71, samples=13\n'
' lat (usec) : 10=36.53%, '
'20=61.98%, 50=1.45%, 100=0.04%\n'
' cpu : usr=34.76%, '
'sys=63.13%, ctx=12125, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2027652,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=141MiB/s (148MB/s), '
'141MiB/s-141MiB/s '
'(148MB/s-148MB/s), io=990MiB '
'(1038MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2908270/0, '
'merge=0/0, ticks=15201/0, '
'in_queue=15201, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.output'),
'state': {'begin': 1666745109.687747,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7985000610351562,
'end': 1666745110.486247,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.state')},
'cmd_268': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.output'),
'state': {'begin': 1666745111.489625,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.783801794052124,
'end': 1666745112.2734268,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.state')},
'cmd_269': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.output'),
'state': {'begin': 1666745112.2754273,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.081075668334961,
'end': 1666745123.356503,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.state')},
'cmd_27': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751053,\n'
' "timestamp_ms" : '
'1666751053526,\n'
' "time" : "Wed Oct 26 04:24:13 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 439462400,\n'
' "io_kbytes" : 429162,\n'
' "bw_bytes" : 62771375,\n'
' "bw" : 61300,\n'
' "iops" : 122600.199971,\n'
' "runtime" : 7001,\n'
' "total_ios" : 858324,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 701,\n'
' "max" : 13997,\n'
' "mean" : 918.099893,\n'
' "stddev" : 66.379994,\n'
' "N" : 858325\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 551,\n'
' "max" : 101871,\n'
' "mean" : 6972.345603,\n'
' "stddev" : '
'1116.060535,\n'
' "N" : 858324,\n'
' "percentile" : {\n'
' "1.000000" : 6368,\n'
' "5.000000" : 6496,\n'
' "10.000000" : 6560,\n'
' "20.000000" : 6688,\n'
' "30.000000" : 6752,\n'
' "40.000000" : 6752,\n'
' "50.000000" : 6816,\n'
' "60.000000" : 6944,\n'
' "70.000000" : 6944,\n'
' "80.000000" : 7008,\n'
' "90.000000" : 7136,\n'
' "95.000000" : 7264,\n'
' "99.000000" : 12864,\n'
' "99.500000" : 13632,\n'
' "99.900000" : 16512,\n'
' "99.950000" : 16768,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7013,\n'
' "max" : 102792,\n'
' "mean" : 7890.445503,\n'
' "stddev" : '
'1119.494292,\n'
' "N" : 858324\n'
' },\n'
' "bw_min" : 61255,\n'
' "bw_max" : 61417,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'61317.071429,\n'
' "bw_dev" : 43.492642,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 122510,\n'
' "iops_max" : 122834,\n'
' "iops_mean" : '
'122634.214286,\n'
' "iops_stddev" : '
'86.918334,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.301574,\n'
' "20" : 1.679669,\n'
' "50" : 0.010000,\n'
' "100" : 0.014563,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=139019: Wed Oct 26 04:24:13 '
'2022\n'
' read: IOPS=123k, BW=59.9MiB/s '
'(62.8MB/s)(419MiB/7001msec)\n'
' slat (nsec): min=701, '
'max=13997, avg=918.10, '
'stdev=66.38\n'
' clat (nsec): min=551, '
'max=101871, avg=6972.35, '
'stdev=1116.06\n'
' lat (usec): min=7, max=102, '
'avg= 7.89, stdev= 1.12\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6368], 5.00th=[ '
'6496], 10.00th=[ 6560], 20.00th=[ '
'6688],\n'
' | 30.00th=[ 6752], 40.00th=[ '
'6752], 50.00th=[ 6816], 60.00th=[ '
'6944],\n'
' | 70.00th=[ 6944], 80.00th=[ '
'7008], 90.00th=[ 7136], 95.00th=[ '
'7264],\n'
' | 99.00th=[12864], '
'99.50th=[13632], 99.90th=[16512], '
'99.95th=[16768],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=61255, '
'max=61417, per=100.00%, '
'avg=61317.07, stdev=43.49, '
'samples=14\n'
' iops : min=122510, '
'max=122834, avg=122634.21, '
'stdev=86.92, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 10=98.30%, '
'20=1.68%, 50=0.01%, 100=0.01%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=858324,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=59.9MiB/s (62.8MB/s), '
'59.9MiB/s-59.9MiB/s '
'(62.8MB/s-62.8MB/s), io=419MiB '
'(439MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.output'),
'state': {'begin': 1666743964.3977723,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7926888465881348,
'end': 1666743965.1904612,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.state')},
'cmd_270': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752212,\n'
' "timestamp_ms" : '
'1666752212538,\n'
' "time" : "Wed Oct 26 04:43:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1037223424,\n'
' "io_kbytes" : 1012913,\n'
' "bw_bytes" : 148153610,\n'
' "bw" : 144681,\n'
' "iops" : 289361.948293,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2025823,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1412,\n'
' "max" : 28864,\n'
' "mean" : 2730.817650,\n'
' "stddev" : '
'1328.449286,\n'
' "N" : 2025824\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4679,\n'
' "max" : 134222,\n'
' "mean" : '
'10856.180382,\n'
' "stddev" : '
'2794.572731,\n'
' "N" : 2025826,\n'
' "percentile" : {\n'
' "1.000000" : 9024,\n'
' "5.000000" : 9024,\n'
' "10.000000" : 9152,\n'
' "20.000000" : 9152,\n'
' "30.000000" : 9280,\n'
' "40.000000" : '
'10304,\n'
' "50.000000" : '
'10432,\n'
' "60.000000" : '
'10816,\n'
' "70.000000" : '
'11584,\n'
' "80.000000" : '
'11712,\n'
' "90.000000" : '
'11840,\n'
' "95.000000" : '
'12096,\n'
' "99.000000" : '
'23680,\n'
' "99.500000" : '
'29056,\n'
' "99.900000" : '
'35584,\n'
' "99.950000" : '
'42240,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8135,\n'
' "max" : 137117,\n'
' "mean" : '
'13586.998051,\n'
' "stddev" : '
'2658.191293,\n'
' "N" : 2025826\n'
' },\n'
' "bw_min" : 144341,\n'
' "bw_max" : 145442,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'144773.769231,\n'
' "bw_dev" : 340.786383,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 288682,\n'
' "iops_max" : 290884,\n'
' "iops_mean" : '
'289547.461538,\n'
' "iops_stddev" : '
'681.758341,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 35.828571,\n'
' "sys_cpu" : 62.028571,\n'
' "ctx" : 12148,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 36.357915,\n'
' "20" : 62.091604,\n'
' "50" : 1.503537,\n'
' "100" : 0.046253,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2832626,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 14860,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 14860,\n'
' "util" : 99.085923\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=142565: Wed '
'Oct 26 04:43:32 2022\n'
' read: IOPS=289k, BW=141MiB/s '
'(148MB/s)(989MiB/7001msec)\n'
' slat (nsec): min=1412, '
'max=28864, avg=2730.82, '
'stdev=1328.45\n'
' clat (usec): min=4, max=134, '
'avg=10.86, stdev= 2.79\n'
' lat (usec): min=8, max=137, '
'avg=13.59, stdev= 2.66\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9024], '
'5.00th=[ 9024], 10.00th=[ 9152], '
'20.00th=[ 9152],\n'
' | 30.00th=[ 9280], '
'40.00th=[10304], '
'50.00th=[10432], '
'60.00th=[10816],\n'
' | 70.00th=[11584], '
'80.00th=[11712], '
'90.00th=[11840], '
'95.00th=[12096],\n'
' | 99.00th=[23680], '
'99.50th=[29056], '
'99.90th=[35584], '
'99.95th=[42240],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=144341, '
'max=145442, per=100.00%, '
'avg=144773.77, stdev=340.79, '
'samples=13\n'
' iops : min=288682, '
'max=290884, avg=289547.46, '
'stdev=681.76, samples=13\n'
' lat (usec) : 10=36.36%, '
'20=62.09%, 50=1.50%, 100=0.05%, '
'250=0.01%\n'
' cpu : usr=35.83%, '
'sys=62.03%, ctx=12148, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2025823,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=141MiB/s (148MB/s), '
'141MiB/s-141MiB/s '
'(148MB/s-148MB/s), io=989MiB '
'(1037MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2832626/0, '
'merge=0/0, ticks=14860/0, '
'in_queue=14860, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.output'),
'state': {'begin': 1666745123.3594093,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7903752326965332,
'end': 1666745124.1497846,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.state')},
'cmd_271': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.output'),
'state': {'begin': 1666745126.1821842,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7924859523773193,
'end': 1666745126.9746702,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.state')},
'cmd_272': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.output'),
'state': {'begin': 1666745126.9771647,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_aio_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.554776906967163,
'end': 1666745138.5319417,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.state')},
'cmd_273': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752227,\n'
' "timestamp_ms" : '
'1666752227603,\n'
' "time" : "Wed Oct 26 04:43:47 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1071235584,\n'
' "io_kbytes" : 1046128,\n'
' "bw_bytes" : 153011796,\n'
' "bw" : 149425,\n'
' "iops" : 298850.878446,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2092255,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1583,\n'
' "max" : 29135,\n'
' "mean" : 2702.866753,\n'
' "stddev" : '
'1007.331736,\n'
' "N" : 2092256\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 260,\n'
' "max" : 114535,\n'
' "mean" : 7822.386945,\n'
' "stddev" : '
'3118.577266,\n'
' "N" : 2092257,\n'
' "percentile" : {\n'
' "1.000000" : 3280,\n'
' "5.000000" : 3376,\n'
' "10.000000" : 4512,\n'
' "20.000000" : 4832,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 6688,\n'
' "50.000000" : 7200,\n'
' "60.000000" : 8512,\n'
' "70.000000" : 9792,\n'
' "80.000000" : '
'10048,\n'
' "90.000000" : '
'11072,\n'
' "95.000000" : '
'11456,\n'
' "99.000000" : '
'18560,\n'
' "99.500000" : '
'21376,\n'
' "99.900000" : '
'24960,\n'
' "99.950000" : '
'31872,\n'
' "99.990000" : 63744\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5791,\n'
' "max" : 116228,\n'
' "mean" : '
'10525.253804,\n'
' "stddev" : '
'3235.371221,\n'
' "N" : 2092257\n'
' },\n'
' "bw_min" : 149025,\n'
' "bw_max" : 149805,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'149469.785714,\n'
' "bw_dev" : 290.944134,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 298050,\n'
' "iops_max" : 299606,\n'
' "iops_mean" : '
'298939.571429,\n'
' "iops_stddev" : '
'581.128440,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 43.357143,\n'
' "sys_cpu" : 56.614286,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.967043,\n'
' "2" : 60.186784,\n'
' "4" : 38.846173,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.128904,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.015390,\n'
' "4" : 7.186170,\n'
' "10" : 73.121154,\n'
' "20" : 18.881733,\n'
' "50" : 0.629417,\n'
' "100" : 0.036946,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=142613: Wed '
'Oct 26 04:43:47 2022\n'
' read: IOPS=299k, BW=146MiB/s '
'(153MB/s)(1022MiB/7001msec)\n'
' slat (nsec): min=1583, '
'max=29135, avg=2702.87, '
'stdev=1007.33\n'
' clat (nsec): min=260, '
'max=114535, avg=7822.39, '
'stdev=3118.58\n'
' lat (usec): min=5, max=116, '
'avg=10.53, stdev= 3.24\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3280], '
'5.00th=[ 3376], 10.00th=[ 4512], '
'20.00th=[ 4832],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 6688], 50.00th=[ '
'7200], 60.00th=[ 8512],\n'
' | 70.00th=[ 9792], '
'80.00th=[10048], '
'90.00th=[11072], '
'95.00th=[11456],\n'
' | 99.00th=[18560], '
'99.50th=[21376], '
'99.90th=[24960], '
'99.95th=[31872],\n'
' | 99.99th=[63744]\n'
' bw ( KiB/s): min=149025, '
'max=149805, per=100.00%, '
'avg=149469.79, stdev=290.94, '
'samples=14\n'
' iops : min=298050, '
'max=299606, avg=298939.57, '
'stdev=581.13, samples=14\n'
' lat (nsec) : 500=0.13%, '
'750=0.01%, 1000=0.01%\n'
' lat (usec) : 2=0.02%, '
'4=7.19%, 10=73.12%, 20=18.88%, '
'50=0.63%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=43.36%, '
'sys=56.61%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=1.0%, '
'2=60.2%, 4=38.8%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2092255,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=146MiB/s (153MB/s), '
'146MiB/s-146MiB/s '
'(153MB/s-153MB/s), io=1022MiB '
'(1071MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.output'),
'state': {'begin': 1666745138.535005,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7863025665283203,
'end': 1666745139.3213077,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.state')},
'cmd_274': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.output'),
'state': {'begin': 1666745141.3469138,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7810285091400146,
'end': 1666745142.1279423,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.state')},
'cmd_275': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.output'),
'state': {'begin': 1666745142.130097,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_aio_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.5094473361969,
'end': 1666745153.6395442,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.state')},
'cmd_276': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752242,\n'
' "timestamp_ms" : '
'1666752242721,\n'
' "time" : "Wed Oct 26 04:44:02 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1061995520,\n'
' "io_kbytes" : 1037105,\n'
' "bw_bytes" : 151691975,\n'
' "bw" : 148136,\n'
' "iops" : 296273.103842,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2074208,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1603,\n'
' "max" : 57037,\n'
' "mean" : 2788.654794,\n'
' "stddev" : '
'1082.503450,\n'
' "N" : 2074209\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 241,\n'
' "max" : 157215,\n'
' "mean" : 7732.501443,\n'
' "stddev" : '
'3123.606252,\n'
' "N" : 2074210,\n'
' "percentile" : {\n'
' "1.000000" : 3312,\n'
' "5.000000" : 4512,\n'
' "10.000000" : 4576,\n'
' "20.000000" : 4704,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 6560,\n'
' "50.000000" : 6880,\n'
' "60.000000" : 8256,\n'
' "70.000000" : 9920,\n'
' "80.000000" : '
'10048,\n'
' "90.000000" : '
'11200,\n'
' "95.000000" : '
'11456,\n'
' "99.000000" : '
'18304,\n'
' "99.500000" : '
'21120,\n'
' "99.900000" : '
'25216,\n'
' "99.950000" : '
'31872,\n'
' "99.990000" : 62720\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5721,\n'
' "max" : 158908,\n'
' "mean" : '
'10521.156284,\n'
' "stddev" : '
'3475.499276,\n'
' "N" : 2074210\n'
' },\n'
' "bw_min" : 147810,\n'
' "bw_max" : 148550,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'148173.714286,\n'
' "bw_dev" : 244.282443,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 295620,\n'
' "iops_max" : 297100,\n'
' "iops_mean" : '
'296347.428571,\n'
' "iops_stddev" : '
'488.564887,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 43.757143,\n'
' "sys_cpu" : 56.214286,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.941564,\n'
' "2" : 61.963410,\n'
' "4" : 37.095026,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.128868,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.017645,\n'
' "4" : 4.562320,\n'
' "10" : 70.359482,\n'
' "20" : 24.313714,\n'
' "50" : 0.580125,\n'
' "100" : 0.036978,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=142662: Wed '
'Oct 26 04:44:02 2022\n'
' read: IOPS=296k, BW=145MiB/s '
'(152MB/s)(1013MiB/7001msec)\n'
' slat (nsec): min=1603, '
'max=57037, avg=2788.65, '
'stdev=1082.50\n'
' clat (nsec): min=241, '
'max=157215, avg=7732.50, '
'stdev=3123.61\n'
' lat (usec): min=5, max=158, '
'avg=10.52, stdev= 3.48\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3312], '
'5.00th=[ 4512], 10.00th=[ 4576], '
'20.00th=[ 4704],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 6560], 50.00th=[ '
'6880], 60.00th=[ 8256],\n'
' | 70.00th=[ 9920], '
'80.00th=[10048], '
'90.00th=[11200], '
'95.00th=[11456],\n'
' | 99.00th=[18304], '
'99.50th=[21120], '
'99.90th=[25216], '
'99.95th=[31872],\n'
' | 99.99th=[62720]\n'
' bw ( KiB/s): min=147810, '
'max=148550, per=100.00%, '
'avg=148173.71, stdev=244.28, '
'samples=14\n'
' iops : min=295620, '
'max=297100, avg=296347.43, '
'stdev=488.56, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.13%, 750=0.01%\n'
' lat (usec) : 2=0.02%, '
'4=4.56%, 10=70.36%, 20=24.31%, '
'50=0.58%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=43.76%, '
'sys=56.21%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=0.9%, '
'2=62.0%, 4=37.1%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2074208,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=145MiB/s (152MB/s), '
'145MiB/s-145MiB/s '
'(152MB/s-152MB/s), io=1013MiB '
'(1062MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.output'),
'state': {'begin': 1666745153.6424088,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7868928909301758,
'end': 1666745154.4293017,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.state')},
'cmd_277': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.output'),
'state': {'begin': 1666745156.4354622,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7887165546417236,
'end': 1666745157.2241788,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.state')},
'cmd_278': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.output'),
'state': {'begin': 1666745157.2262728,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_aio_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.493136167526245,
'end': 1666745168.719409,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.state')},
'cmd_279': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752257,\n'
' "timestamp_ms" : '
'1666752257808,\n'
' "time" : "Wed Oct 26 04:44:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1064985600,\n'
' "io_kbytes" : 1040025,\n'
' "bw_bytes" : 152119068,\n'
' "bw" : 148553,\n'
' "iops" : 297107.270390,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2080048,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1593,\n'
' "max" : 32571,\n'
' "mean" : 2739.548837,\n'
' "stddev" : '
'1047.822313,\n'
' "N" : 2080049\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 270,\n'
' "max" : 117951,\n'
' "mean" : 7801.096301,\n'
' "stddev" : '
'3165.549384,\n'
' "N" : 2080050,\n'
' "percentile" : {\n'
' "1.000000" : 3312,\n'
' "5.000000" : 3440,\n'
' "10.000000" : 4576,\n'
' "20.000000" : 4704,\n'
' "30.000000" : 6432,\n'
' "40.000000" : 6560,\n'
' "50.000000" : 6944,\n'
' "60.000000" : 8512,\n'
' "70.000000" : 9920,\n'
' "80.000000" : '
'10048,\n'
' "90.000000" : '
'11200,\n'
' "95.000000" : '
'11456,\n'
' "99.000000" : '
'18304,\n'
' "99.500000" : '
'21120,\n'
' "99.900000" : '
'25472,\n'
' "99.950000" : '
'32128,\n'
' "99.990000" : 64256\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5830,\n'
' "max" : 119664,\n'
' "mean" : '
'10540.645231,\n'
' "stddev" : '
'3389.356523,\n'
' "N" : 2080050\n'
' },\n'
' "bw_min" : 148084,\n'
' "bw_max" : 149034,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'148592.357143,\n'
' "bw_dev" : 285.577043,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 296168,\n'
' "iops_max" : 298068,\n'
' "iops_mean" : '
'297184.714286,\n'
' "iops_stddev" : '
'571.154085,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 41.985714,\n'
' "sys_cpu" : 57.985714,\n'
' "ctx" : 25,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.959641,\n'
' "2" : 61.257433,\n'
' "4" : 37.782926,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.123459,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.015096,\n'
' "4" : 6.310912,\n'
' "10" : 69.856369,\n'
' "20" : 23.058602,\n'
' "50" : 0.596188,\n'
' "100" : 0.038797,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=142712: Wed '
'Oct 26 04:44:17 2022\n'
' read: IOPS=297k, BW=145MiB/s '
'(152MB/s)(1016MiB/7001msec)\n'
' slat (nsec): min=1593, '
'max=32571, avg=2739.55, '
'stdev=1047.82\n'
' clat (nsec): min=270, '
'max=117951, avg=7801.10, '
'stdev=3165.55\n'
' lat (usec): min=5, max=119, '
'avg=10.54, stdev= 3.39\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3312], '
'5.00th=[ 3440], 10.00th=[ 4576], '
'20.00th=[ 4704],\n'
' | 30.00th=[ 6432], '
'40.00th=[ 6560], 50.00th=[ '
'6944], 60.00th=[ 8512],\n'
' | 70.00th=[ 9920], '
'80.00th=[10048], '
'90.00th=[11200], '
'95.00th=[11456],\n'
' | 99.00th=[18304], '
'99.50th=[21120], '
'99.90th=[25472], '
'99.95th=[32128],\n'
' | 99.99th=[64256]\n'
' bw ( KiB/s): min=148084, '
'max=149034, per=100.00%, '
'avg=148592.36, stdev=285.58, '
'samples=14\n'
' iops : min=296168, '
'max=298068, avg=297184.71, '
'stdev=571.15, samples=14\n'
' lat (nsec) : 500=0.12%\n'
' lat (usec) : 2=0.02%, '
'4=6.31%, 10=69.86%, 20=23.06%, '
'50=0.60%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=41.99%, '
'sys=57.99%, ctx=25, majf=0, '
'minf=0\n'
' IO depths : 1=1.0%, '
'2=61.3%, 4=37.8%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2080048,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=145MiB/s (152MB/s), '
'145MiB/s-145MiB/s '
'(152MB/s-152MB/s), io=1016MiB '
'(1065MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.output'),
'state': {'begin': 1666745168.7224128,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7964358329772949,
'end': 1666745169.5188487,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.state')},
'cmd_28': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.output'),
'state': {'begin': 1666743966.1824296,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7796878814697266,
'end': 1666743966.9621174,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.state')},
'cmd_280': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.output'),
'state': {'begin': 1666745171.658184,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7816324234008789,
'end': 1666745172.4398165,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.state')},
'cmd_281': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.output'),
'state': {'begin': 1666745172.4418705,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.521850824356079,
'end': 1666745183.9637213,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.state')},
'cmd_282': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752273,\n'
' "timestamp_ms" : '
'1666752273038,\n'
' "time" : "Wed Oct 26 04:44:33 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1012254720,\n'
' "io_kbytes" : 988530,\n'
' "bw_bytes" : 144587161,\n'
' "bw" : 141198,\n'
' "iops" : 282396.514784,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1977058,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1603,\n'
' "max" : 63850,\n'
' "mean" : 2849.242207,\n'
' "stddev" : '
'1275.330036,\n'
' "N" : 1977059\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 641,\n'
' "max" : 100899,\n'
' "mean" : 8513.515201,\n'
' "stddev" : '
'2759.933150,\n'
' "N" : 1977060,\n'
' "percentile" : {\n'
' "1.000000" : 4832,\n'
' "5.000000" : 4960,\n'
' "10.000000" : 5088,\n'
' "20.000000" : 6880,\n'
' "30.000000" : 7456,\n'
' "40.000000" : 7584,\n'
' "50.000000" : 7648,\n'
' "60.000000" : 9280,\n'
' "70.000000" : 9408,\n'
' "80.000000" : 9536,\n'
' "90.000000" : '
'11328,\n'
' "95.000000" : '
'12224,\n'
' "99.000000" : '
'18560,\n'
' "99.500000" : '
'21120,\n'
' "99.900000" : '
'25216,\n'
' "99.950000" : '
'31872,\n'
' "99.990000" : 63744\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6052,\n'
' "max" : 102622,\n'
' "mean" : '
'11362.757426,\n'
' "stddev" : '
'3379.702263,\n'
' "N" : 1977060\n'
' },\n'
' "bw_min" : 140715,\n'
' "bw_max" : 141758,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'141236.428571,\n'
' "bw_dev" : 312.103216,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 281430,\n'
' "iops_max" : 283516,\n'
' "iops_mean" : '
'282472.928571,\n'
' "iops_stddev" : '
'624.256415,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 43.642857,\n'
' "sys_cpu" : 56.328571,\n'
' "ctx" : 24,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.877010,\n'
' "2" : 56.256974,\n'
' "4" : 42.866016,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.053918,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.015680,\n'
' "4" : 0.791783,\n'
' "10" : 82.251962,\n'
' "20" : 16.201093,\n'
' "50" : 0.643279,\n'
' "100" : 0.037986,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142761: Wed '
'Oct 26 04:44:33 2022\n'
' read: IOPS=282k, BW=138MiB/s '
'(145MB/s)(965MiB/7001msec)\n'
' slat (nsec): min=1603, '
'max=63850, avg=2849.24, '
'stdev=1275.33\n'
' clat (nsec): min=641, '
'max=100899, avg=8513.52, '
'stdev=2759.93\n'
' lat (usec): min=6, max=102, '
'avg=11.36, stdev= 3.38\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4832], '
'5.00th=[ 4960], 10.00th=[ 5088], '
'20.00th=[ 6880],\n'
' | 30.00th=[ 7456], '
'40.00th=[ 7584], 50.00th=[ '
'7648], 60.00th=[ 9280],\n'
' | 70.00th=[ 9408], '
'80.00th=[ 9536], '
'90.00th=[11328], '
'95.00th=[12224],\n'
' | 99.00th=[18560], '
'99.50th=[21120], '
'99.90th=[25216], '
'99.95th=[31872],\n'
' | 99.99th=[63744]\n'
' bw ( KiB/s): min=140715, '
'max=141758, per=100.00%, '
'avg=141236.43, stdev=312.10, '
'samples=14\n'
' iops : min=281430, '
'max=283516, avg=282472.93, '
'stdev=624.26, samples=14\n'
' lat (nsec) : 750=0.05%, '
'1000=0.01%\n'
' lat (usec) : 2=0.02%, '
'4=0.79%, 10=82.25%, 20=16.20%, '
'50=0.64%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=43.64%, '
'sys=56.33%, ctx=24, majf=0, '
'minf=0\n'
' IO depths : 1=0.9%, '
'2=56.3%, 4=42.9%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1977058,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=138MiB/s (145MB/s), '
'138MiB/s-138MiB/s '
'(145MB/s-145MB/s), io=965MiB '
'(1012MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.output'),
'state': {'begin': 1666745183.9666882,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7833597660064697,
'end': 1666745184.750048,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.state')},
'cmd_283': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.output'),
'state': {'begin': 1666745186.770997,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7856574058532715,
'end': 1666745187.5566545,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.state')},
'cmd_284': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.output'),
'state': {'begin': 1666745187.558708,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.544718265533447,
'end': 1666745199.1034262,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.state')},
'cmd_285': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752288,\n'
' "timestamp_ms" : '
'1666752288178,\n'
' "time" : "Wed Oct 26 04:44:48 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1009187328,\n'
' "io_kbytes" : 985534,\n'
' "bw_bytes" : 144149025,\n'
' "bw" : 140770,\n'
' "iops" : 281540.494215,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1971065,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1613,\n'
' "max" : 36499,\n'
' "mean" : 2864.161118,\n'
' "stddev" : '
'1277.898171,\n'
' "N" : 1971066\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 641,\n'
' "max" : 123301,\n'
' "mean" : 8534.661721,\n'
' "stddev" : '
'2698.421230,\n'
' "N" : 1971065,\n'
' "percentile" : {\n'
' "1.000000" : 4896,\n'
' "5.000000" : 5024,\n'
' "10.000000" : 5152,\n'
' "20.000000" : 6944,\n'
' "30.000000" : 7456,\n'
' "40.000000" : 7584,\n'
' "50.000000" : 7712,\n'
' "60.000000" : 9280,\n'
' "70.000000" : 9408,\n'
' "80.000000" : 9536,\n'
' "90.000000" : '
'10944,\n'
' "95.000000" : '
'12224,\n'
' "99.000000" : '
'18304,\n'
' "99.500000" : '
'21120,\n'
' "99.900000" : '
'25472,\n'
' "99.950000" : '
'32384,\n'
' "99.990000" : 63744\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6042,\n'
' "max" : 127870,\n'
' "mean" : '
'11398.823449,\n'
' "stddev" : '
'3337.911286,\n'
' "N" : 1971065\n'
' },\n'
' "bw_min" : 139935,\n'
' "bw_max" : 141743,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'140813.428571,\n'
' "bw_dev" : 451.996207,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 279870,\n'
' "iops_max" : 283486,\n'
' "iops_mean" : '
'281626.928571,\n'
' "iops_stddev" : '
'904.069642,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 44.785714,\n'
' "sys_cpu" : 55.185714,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.869327,\n'
' "2" : 55.711506,\n'
' "4" : 43.419167,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.051191,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.016691,\n'
' "4" : 0.692570,\n'
' "10" : 83.599323,\n'
' "20" : 14.998186,\n'
' "50" : 0.595871,\n'
' "100" : 0.041500,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142811: Wed '
'Oct 26 04:44:48 2022\n'
' read: IOPS=282k, BW=137MiB/s '
'(144MB/s)(962MiB/7001msec)\n'
' slat (nsec): min=1613, '
'max=36499, avg=2864.16, '
'stdev=1277.90\n'
' clat (nsec): min=641, '
'max=123301, avg=8534.66, '
'stdev=2698.42\n'
' lat (usec): min=6, max=127, '
'avg=11.40, stdev= 3.34\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4896], '
'5.00th=[ 5024], 10.00th=[ 5152], '
'20.00th=[ 6944],\n'
' | 30.00th=[ 7456], '
'40.00th=[ 7584], 50.00th=[ '
'7712], 60.00th=[ 9280],\n'
' | 70.00th=[ 9408], '
'80.00th=[ 9536], '
'90.00th=[10944], '
'95.00th=[12224],\n'
' | 99.00th=[18304], '
'99.50th=[21120], '
'99.90th=[25472], '
'99.95th=[32384],\n'
' | 99.99th=[63744]\n'
' bw ( KiB/s): min=139935, '
'max=141743, per=100.00%, '
'avg=140813.43, stdev=452.00, '
'samples=14\n'
' iops : min=279870, '
'max=283486, avg=281626.93, '
'stdev=904.07, samples=14\n'
' lat (nsec) : 750=0.05%, '
'1000=0.01%\n'
' lat (usec) : 2=0.02%, '
'4=0.69%, 10=83.60%, 20=15.00%, '
'50=0.60%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=44.79%, '
'sys=55.19%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=0.9%, '
'2=55.7%, 4=43.4%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1971065,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=137MiB/s (144MB/s), '
'137MiB/s-137MiB/s '
'(144MB/s-144MB/s), io=962MiB '
'(1009MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.output'),
'state': {'begin': 1666745199.106262,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7916183471679688,
'end': 1666745199.8978803,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.state')},
'cmd_286': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.output'),
'state': {'begin': 1666745201.9087136,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7794864177703857,
'end': 1666745202.6882,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.state')},
'cmd_287': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.output'),
'state': {'begin': 1666745202.6904583,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.516079425811768,
'end': 1666745214.2065377,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.state')},
'cmd_288': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752303,\n'
' "timestamp_ms" : '
'1666752303292,\n'
' "time" : "Wed Oct 26 04:45:03 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1011238912,\n'
' "io_kbytes" : 987538,\n'
' "bw_bytes" : 144442067,\n'
' "bw" : 141056,\n'
' "iops" : 282112.841023,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1975072,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1613,\n'
' "max" : 61526,\n'
' "mean" : 2852.995997,\n'
' "stddev" : '
'1262.832611,\n'
' "N" : 1975073\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 631,\n'
' "max" : 118362,\n'
' "mean" : 8518.367863,\n'
' "stddev" : '
'2664.843579,\n'
' "N" : 1975072,\n'
' "percentile" : {\n'
' "1.000000" : 4832,\n'
' "5.000000" : 4960,\n'
' "10.000000" : 5088,\n'
' "20.000000" : 6880,\n'
' "30.000000" : 7456,\n'
' "40.000000" : 7584,\n'
' "50.000000" : 7712,\n'
' "60.000000" : 9280,\n'
' "70.000000" : 9408,\n'
' "80.000000" : 9536,\n'
' "90.000000" : '
'10944,\n'
' "95.000000" : '
'12224,\n'
' "99.000000" : '
'18304,\n'
' "99.500000" : '
'20864,\n'
' "99.900000" : '
'24960,\n'
' "99.950000" : '
'30592,\n'
' "99.990000" : 62208\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6021,\n'
' "max" : 120075,\n'
' "mean" : '
'11371.363823,\n'
' "stddev" : '
'3297.605000,\n'
' "N" : 1975072\n'
' },\n'
' "bw_min" : 140440,\n'
' "bw_max" : 142061,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'141095.714286,\n'
' "bw_dev" : 440.312377,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 280880,\n'
' "iops_max" : 284122,\n'
' "iops_mean" : '
'282191.428571,\n'
' "iops_stddev" : '
'880.624753,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 43.471429,\n'
' "sys_cpu" : 56.500000,\n'
' "ctx" : 20,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.861437,\n'
' "2" : 55.847635,\n'
' "4" : 43.290928,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.052150,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.018075,\n'
' "4" : 0.862348,\n'
' "10" : 82.898092,\n'
' "20" : 15.549256,\n'
' "50" : 0.580789,\n'
' "100" : 0.035391,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142860: Wed '
'Oct 26 04:45:03 2022\n'
' read: IOPS=282k, BW=138MiB/s '
'(144MB/s)(964MiB/7001msec)\n'
' slat (nsec): min=1613, '
'max=61526, avg=2853.00, '
'stdev=1262.83\n'
' clat (nsec): min=631, '
'max=118362, avg=8518.37, '
'stdev=2664.84\n'
' lat (usec): min=6, max=120, '
'avg=11.37, stdev= 3.30\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4832], '
'5.00th=[ 4960], 10.00th=[ 5088], '
'20.00th=[ 6880],\n'
' | 30.00th=[ 7456], '
'40.00th=[ 7584], 50.00th=[ '
'7712], 60.00th=[ 9280],\n'
' | 70.00th=[ 9408], '
'80.00th=[ 9536], '
'90.00th=[10944], '
'95.00th=[12224],\n'
' | 99.00th=[18304], '
'99.50th=[20864], '
'99.90th=[24960], '
'99.95th=[30592],\n'
' | 99.99th=[62208]\n'
' bw ( KiB/s): min=140440, '
'max=142061, per=100.00%, '
'avg=141095.71, stdev=440.31, '
'samples=14\n'
' iops : min=280880, '
'max=284122, avg=282191.43, '
'stdev=880.62, samples=14\n'
' lat (nsec) : 750=0.05%, '
'1000=0.01%\n'
' lat (usec) : 2=0.02%, '
'4=0.86%, 10=82.90%, 20=15.55%, '
'50=0.58%\n'
' lat (usec) : 100=0.04%, '
'250=0.01%\n'
' cpu : usr=43.47%, '
'sys=56.50%, ctx=20, majf=0, '
'minf=0\n'
' IO depths : 1=0.9%, '
'2=55.8%, 4=43.3%, 8=0.0%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1975072,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=138MiB/s (144MB/s), '
'138MiB/s-138MiB/s '
'(144MB/s-144MB/s), io=964MiB '
'(1011MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.output'),
'state': {'begin': 1666745214.209567,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7944097518920898,
'end': 1666745215.0039768,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.state')},
'cmd_289': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.output'),
'state': {'begin': 1666745215.9990199,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7896780967712402,
'end': 1666745216.788698,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.state')},
'cmd_29': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.output'),
'state': {'begin': 1666743966.9641044,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.041018009185791,
'end': 1666743978.0051224,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.state')},
'cmd_290': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.output'),
'state': {'begin': 1666745216.7908251,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.089628458023071,
'end': 1666745227.8804536,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.state')},
'cmd_291': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752317,\n'
' "timestamp_ms" : '
'1666752317061,\n'
' "time" : "Wed Oct 26 04:45:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 995397120,\n'
' "io_kbytes" : 972067,\n'
' "bw_bytes" : 142179277,\n'
' "bw" : 138846,\n'
' "iops" : 277693.329524,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1944131,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1472,\n'
' "max" : 23805,\n'
' "mean" : 2275.900375,\n'
' "stddev" : '
'900.413544,\n'
' "N" : 1944132\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4949,\n'
' "max" : 133701,\n'
' "mean" : '
'11331.964851,\n'
' "stddev" : '
'2845.799578,\n'
' "N" : 1944134,\n'
' "percentile" : {\n'
' "1.000000" : 9408,\n'
' "5.000000" : 9536,\n'
' "10.000000" : 9536,\n'
' "20.000000" : 9664,\n'
' "30.000000" : 9664,\n'
' "40.000000" : 9792,\n'
' "50.000000" : '
'11072,\n'
' "60.000000" : '
'12096,\n'
' "70.000000" : '
'12224,\n'
' "80.000000" : '
'12352,\n'
' "90.000000" : '
'12352,\n'
' "95.000000" : '
'12608,\n'
' "99.000000" : '
'23680,\n'
' "99.500000" : '
'28288,\n'
' "99.900000" : '
'36096,\n'
' "99.950000" : '
'40192,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7484,\n'
' "max" : 136586,\n'
' "mean" : '
'13607.865899,\n'
' "stddev" : '
'2632.972492,\n'
' "N" : 1944134\n'
' },\n'
' "bw_min" : 138554,\n'
' "bw_max" : 139440,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'138885.857143,\n'
' "bw_dev" : 275.370200,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 277108,\n'
' "iops_max" : 278880,\n'
' "iops_mean" : '
'277771.785714,\n'
' "iops_stddev" : '
'550.705586,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 36.685714,\n'
' "sys_cpu" : 61.342857,\n'
' "ctx" : 11813,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 45.621154,\n'
' "20" : 52.590952,\n'
' "50" : 1.747002,\n'
' "100" : 0.039349,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142899: Wed '
'Oct 26 04:45:17 2022\n'
' read: IOPS=278k, BW=136MiB/s '
'(142MB/s)(949MiB/7001msec)\n'
' slat (nsec): min=1472, '
'max=23805, avg=2275.90, '
'stdev=900.41\n'
' clat (usec): min=4, max=133, '
'avg=11.33, stdev= 2.85\n'
' lat (usec): min=7, max=136, '
'avg=13.61, stdev= 2.63\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9408], '
'5.00th=[ 9536], 10.00th=[ 9536], '
'20.00th=[ 9664],\n'
' | 30.00th=[ 9664], '
'40.00th=[ 9792], '
'50.00th=[11072], '
'60.00th=[12096],\n'
' | 70.00th=[12224], '
'80.00th=[12352], '
'90.00th=[12352], '
'95.00th=[12608],\n'
' | 99.00th=[23680], '
'99.50th=[28288], '
'99.90th=[36096], '
'99.95th=[40192],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=138554, '
'max=139440, per=100.00%, '
'avg=138885.86, stdev=275.37, '
'samples=14\n'
' iops : min=277108, '
'max=278880, avg=277771.79, '
'stdev=550.71, samples=14\n'
' lat (usec) : 10=45.62%, '
'20=52.59%, 50=1.75%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=36.69%, '
'sys=61.34%, ctx=11813, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1944131,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=136MiB/s (142MB/s), '
'136MiB/s-136MiB/s '
'(142MB/s-142MB/s), io=949MiB '
'(995MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.output'),
'state': {'begin': 1666745227.883581,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8156938552856445,
'end': 1666745228.6992748,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.state')},
'cmd_292': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.output'),
'state': {'begin': 1666745229.7561324,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.8139240741729736,
'end': 1666745230.5700564,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.state')},
'cmd_293': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.output'),
'state': {'begin': 1666745230.5723808,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.087549924850464,
'end': 1666745241.6599307,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.state')},
'cmd_294': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752330,\n'
' "timestamp_ms" : '
'1666752330841,\n'
' "time" : "Wed Oct 26 04:45:30 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 998190080,\n'
' "io_kbytes" : 974795,\n'
' "bw_bytes" : 142578214,\n'
' "bw" : 139236,\n'
' "iops" : 278472.503928,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1949586,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1462,\n'
' "max" : 34996,\n'
' "mean" : 2286.772988,\n'
' "stddev" : '
'911.175884,\n'
' "N" : 1949587\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 501,\n'
' "max" : 134612,\n'
' "mean" : '
'11299.563884,\n'
' "stddev" : '
'2830.245084,\n'
' "N" : 1949589,\n'
' "percentile" : {\n'
' "1.000000" : 9408,\n'
' "5.000000" : 9536,\n'
' "10.000000" : 9536,\n'
' "20.000000" : 9536,\n'
' "30.000000" : 9664,\n'
' "40.000000" : 9664,\n'
' "50.000000" : '
'10944,\n'
' "60.000000" : '
'12096,\n'
' "70.000000" : '
'12224,\n'
' "80.000000" : '
'12224,\n'
' "90.000000" : '
'12352,\n'
' "95.000000" : '
'12608,\n'
' "99.000000" : '
'23680,\n'
' "99.500000" : '
'28288,\n'
' "99.900000" : '
'36096,\n'
' "99.950000" : '
'41216,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8206,\n'
' "max" : 138840,\n'
' "mean" : '
'13586.337374,\n'
' "stddev" : '
'2615.938007,\n'
' "N" : 1949589\n'
' },\n'
' "bw_min" : 138799,\n'
' "bw_max" : 139789,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'139298.538462,\n'
' "bw_dev" : 326.805145,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 277600,\n'
' "iops_max" : 279578,\n'
' "iops_mean" : '
'278597.230769,\n'
' "iops_stddev" : '
'653.355717,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 38.457143,\n'
' "sys_cpu" : 59.557143,\n'
' "ctx" : 11881,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 45.654565,\n'
' "20" : 52.723501,\n'
' "50" : 1.577668,\n'
' "100" : 0.043599,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142939: Wed '
'Oct 26 04:45:30 2022\n'
' read: IOPS=278k, BW=136MiB/s '
'(143MB/s)(952MiB/7001msec)\n'
' slat (nsec): min=1462, '
'max=34996, avg=2286.77, '
'stdev=911.18\n'
' clat (nsec): min=501, '
'max=134612, avg=11299.56, '
'stdev=2830.25\n'
' lat (usec): min=8, max=138, '
'avg=13.59, stdev= 2.62\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9408], '
'5.00th=[ 9536], 10.00th=[ 9536], '
'20.00th=[ 9536],\n'
' | 30.00th=[ 9664], '
'40.00th=[ 9664], '
'50.00th=[10944], '
'60.00th=[12096],\n'
' | 70.00th=[12224], '
'80.00th=[12224], '
'90.00th=[12352], '
'95.00th=[12608],\n'
' | 99.00th=[23680], '
'99.50th=[28288], '
'99.90th=[36096], '
'99.95th=[41216],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=138799, '
'max=139789, per=100.00%, '
'avg=139298.54, stdev=326.81, '
'samples=13\n'
' iops : min=277600, '
'max=279578, avg=278597.23, '
'stdev=653.36, samples=13\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 10=45.65%, '
'20=52.72%, 50=1.58%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=38.46%, '
'sys=59.56%, ctx=11881, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1949586,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=136MiB/s (143MB/s), '
'136MiB/s-136MiB/s '
'(143MB/s-143MB/s), io=952MiB '
'(998MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.output'),
'state': {'begin': 1666745241.6631203,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7908663749694824,
'end': 1666745242.4539866,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.state')},
'cmd_295': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.output'),
'state': {'begin': 1666745243.446813,
'cmd': 'rm '
'/tmp/fio-output_512_4_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7882969379425049,
'end': 1666745244.23511,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.state')},
'cmd_296': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.output'),
'state': {'begin': 1666745244.2373972,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="4" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_4_libaio-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.09430480003357,
'end': 1666745255.331702,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.state')},
'cmd_297': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=4\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752344,\n'
' "timestamp_ms" : '
'1666752344512,\n'
' "time" : "Wed Oct 26 04:45:44 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "4",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1003818496,\n'
' "io_kbytes" : 980291,\n'
' "bw_bytes" : 143382159,\n'
' "bw" : 140021,\n'
' "iops" : 280042.708185,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1960579,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1462,\n'
' "max" : 27762,\n'
' "mean" : 2373.694871,\n'
' "stddev" : '
'1047.956777,\n'
' "N" : 1960580\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4919,\n'
' "max" : 134282,\n'
' "mean" : '
'11239.872907,\n'
' "stddev" : '
'2798.284062,\n'
' "N" : 1960582,\n'
' "percentile" : {\n'
' "1.000000" : 9408,\n'
' "5.000000" : 9408,\n'
' "10.000000" : 9408,\n'
' "20.000000" : 9536,\n'
' "30.000000" : 9536,\n'
' "40.000000" : 9664,\n'
' "50.000000" : '
'10944,\n'
' "60.000000" : '
'11968,\n'
' "70.000000" : '
'12096,\n'
' "80.000000" : '
'12224,\n'
' "90.000000" : '
'12352,\n'
' "95.000000" : '
'12608,\n'
' "99.000000" : '
'23680,\n'
' "99.500000" : '
'28288,\n'
' "99.900000" : '
'36096,\n'
' "99.950000" : '
'40192,\n'
' "99.990000" : 70144\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7174,\n'
' "max" : 138400,\n'
' "mean" : '
'13613.568208,\n'
' "stddev" : '
'2576.989435,\n'
' "N" : 1960582\n'
' },\n'
' "bw_min" : 139649,\n'
' "bw_max" : 140398,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'140066.571429,\n'
' "bw_dev" : 220.341593,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 279298,\n'
' "iops_max" : 280796,\n'
' "iops_mean" : '
'280133.214286,\n'
' "iops_stddev" : '
'440.635935,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 38.142857,\n'
' "sys_cpu" : 59.885714,\n'
' "ctx" : 11851,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 45.917150,\n'
' "20" : 52.538204,\n'
' "50" : 1.504913,\n'
' "100" : 0.039019,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 4,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=142978: Wed '
'Oct 26 04:45:44 2022\n'
' read: IOPS=280k, BW=137MiB/s '
'(143MB/s)(957MiB/7001msec)\n'
' slat (nsec): min=1462, '
'max=27762, avg=2373.69, '
'stdev=1047.96\n'
' clat (usec): min=4, max=134, '
'avg=11.24, stdev= 2.80\n'
' lat (usec): min=7, max=138, '
'avg=13.61, stdev= 2.58\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9408], '
'5.00th=[ 9408], 10.00th=[ 9408], '
'20.00th=[ 9536],\n'
' | 30.00th=[ 9536], '
'40.00th=[ 9664], '
'50.00th=[10944], '
'60.00th=[11968],\n'
' | 70.00th=[12096], '
'80.00th=[12224], '
'90.00th=[12352], '
'95.00th=[12608],\n'
' | 99.00th=[23680], '
'99.50th=[28288], '
'99.90th=[36096], '
'99.95th=[40192],\n'
' | 99.99th=[70144]\n'
' bw ( KiB/s): min=139649, '
'max=140398, per=100.00%, '
'avg=140066.57, stdev=220.34, '
'samples=14\n'
' iops : min=279298, '
'max=280796, avg=280133.21, '
'stdev=440.64, samples=14\n'
' lat (usec) : 10=45.92%, '
'20=52.54%, 50=1.50%, 100=0.04%, '
'250=0.01%\n'
' cpu : usr=38.14%, '
'sys=59.89%, ctx=11851, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1960579,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=4\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=137MiB/s (143MB/s), '
'137MiB/s-137MiB/s '
'(143MB/s-143MB/s), io=957MiB '
'(1004MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.output'),
'state': {'begin': 1666745255.334885,
'cmd': 'cat '
'/tmp/fio-output_512_4_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7952744960784912,
'end': 1666745256.1301594,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.state')},
'cmd_298': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.output'),
'state': {'begin': 1666745257.1318786,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.780454158782959,
'end': 1666745257.9123328,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.state')},
'cmd_299': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.output'),
'state': {'begin': 1666745257.9144704,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.047147750854492,
'end': 1666745268.9616182,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.state')},
'cmd_30': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751067,\n'
' "timestamp_ms" : '
'1666751067207,\n'
' "time" : "Wed Oct 26 04:24:27 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 282276352,\n'
' "io_kbytes" : 275660,\n'
' "bw_bytes" : 40319433,\n'
' "bw" : 39374,\n'
' "iops" : 78748.893015,\n'
' "runtime" : 7001,\n'
' "total_ios" : 551321,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2504,\n'
' "max" : 49713,\n'
' "mean" : 2754.340312,\n'
' "stddev" : 211.568405,\n'
' "N" : 551321\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 200,\n'
' "max" : 99306,\n'
' "mean" : 9273.575117,\n'
' "stddev" : '
'1277.471589,\n'
' "N" : 551321,\n'
' "percentile" : {\n'
' "1.000000" : 8896,\n'
' "5.000000" : 8896,\n'
' "10.000000" : 9024,\n'
' "20.000000" : 9024,\n'
' "30.000000" : 9024,\n'
' "40.000000" : 9024,\n'
' "50.000000" : 9152,\n'
' "60.000000" : 9152,\n'
' "70.000000" : 9152,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9280,\n'
' "95.000000" : 9536,\n'
' "99.000000" : 15168,\n'
' "99.500000" : 15808,\n'
' "99.900000" : 19840,\n'
' "99.950000" : 26240,\n'
' "99.990000" : 56064\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9197,\n'
' "max" : 102102,\n'
' "mean" : 12027.915429,\n'
' "stddev" : '
'1296.149259,\n'
' "N" : 551321\n'
' },\n'
' "bw_min" : 39234,\n'
' "bw_max" : 39603,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'39390.142857,\n'
' "bw_dev" : 106.081142,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 78468,\n'
' "iops_max" : 79206,\n'
' "iops_mean" : '
'78780.357143,\n'
' "iops_stddev" : '
'212.094176,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 12.414286,\n'
' "sys_cpu" : 48.628571,\n'
' "ctx" : 551215,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.031312,\n'
' "20" : 2.883982,\n'
' "50" : 0.057317,\n'
' "100" : 0.019408,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 793290,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5015,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5015,\n'
' "util" : 99.049128\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=139058: Wed '
'Oct 26 04:24:27 2022\n'
' read: IOPS=78.7k, BW=38.5MiB/s '
'(40.3MB/s)(269MiB/7001msec)\n'
' slat (nsec): min=2504, '
'max=49713, avg=2754.34, '
'stdev=211.57\n'
' clat (nsec): min=200, '
'max=99306, avg=9273.58, '
'stdev=1277.47\n'
' lat (usec): min=9, max=102, '
'avg=12.03, stdev= 1.30\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8896], 5.00th=[ '
'8896], 10.00th=[ 9024], 20.00th=[ '
'9024],\n'
' | 30.00th=[ 9024], 40.00th=[ '
'9024], 50.00th=[ 9152], 60.00th=[ '
'9152],\n'
' | 70.00th=[ 9152], 80.00th=[ '
'9152], 90.00th=[ 9280], 95.00th=[ '
'9536],\n'
' | 99.00th=[15168], '
'99.50th=[15808], 99.90th=[19840], '
'99.95th=[26240],\n'
' | 99.99th=[56064]\n'
' bw ( KiB/s): min=39234, '
'max=39603, per=100.00%, '
'avg=39390.14, stdev=106.08, '
'samples=14\n'
' iops : min=78468, '
'max=79206, avg=78780.36, '
'stdev=212.09, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 10=97.03%, '
'20=2.88%, 50=0.06%, 100=0.02%\n'
' cpu : usr=12.41%, '
'sys=48.63%, ctx=551215, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=551321,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=38.5MiB/s (40.3MB/s), '
'38.5MiB/s-38.5MiB/s '
'(40.3MB/s-40.3MB/s), io=269MiB '
'(282MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=793290/0, '
'merge=0/0, ticks=5015/0, '
'in_queue=5015, util=99.05%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.output'),
'state': {'begin': 1666743978.008007,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7910122871398926,
'end': 1666743978.7990193,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.state')},
'cmd_300': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752358,\n'
' "timestamp_ms" : '
'1666752358142,\n'
' "time" : "Wed Oct 26 04:45:58 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1195644416,\n'
' "io_kbytes" : 1167621,\n'
' "bw_bytes" : 170781947,\n'
' "bw" : 166779,\n'
' "iops" : 333557.491787,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2335236,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1152,\n'
' "max" : 19917,\n'
' "mean" : 2244.668577,\n'
' "stddev" : '
'1068.788020,\n'
' "N" : 2335236\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 145854,\n'
' "mean" : '
'21249.490787,\n'
' "stddev" : '
'3863.814320,\n'
' "N" : 2335243,\n'
' "percentile" : {\n'
' "1.000000" : 13760,\n'
' "5.000000" : 18816,\n'
' "10.000000" : '
'19072,\n'
' "20.000000" : '
'19328,\n'
' "30.000000" : '
'20352,\n'
' "40.000000" : '
'20608,\n'
' "50.000000" : '
'20608,\n'
' "60.000000" : '
'20864,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21888,\n'
' "90.000000" : '
'22656,\n'
' "95.000000" : '
'24960,\n'
' "99.000000" : '
'41728,\n'
' "99.500000" : '
'44288,\n'
' "99.900000" : '
'56064,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 82432\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6612,\n'
' "max" : 147096,\n'
' "mean" : '
'23494.157402,\n'
' "stddev" : '
'3821.091989,\n'
' "N" : 2335243\n'
' },\n'
' "bw_min" : 166104,\n'
' "bw_max" : 167290,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'166794.153846,\n'
' "bw_dev" : 286.537329,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 332210,\n'
' "iops_max" : 334580,\n'
' "iops_mean" : '
'333588.461538,\n'
' "iops_stddev" : '
'572.567989,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.142857,\n'
' "sys_cpu" : 83.414286,\n'
' "ctx" : 12659,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999957,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.016915,\n'
' "20" : 22.109457,\n'
' "50" : 77.740494,\n'
' "100" : 0.130993,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=143018: Wed Oct 26 04:45:58 '
'2022\n'
' read: IOPS=334k, BW=163MiB/s '
'(171MB/s)(1140MiB/7001msec)\n'
' slat (nsec): min=1152, '
'max=19917, avg=2244.67, '
'stdev=1068.79\n'
' clat (nsec): min=80, '
'max=145854, avg=21249.49, '
'stdev=3863.81\n'
' lat (usec): min=6, max=147, '
'avg=23.49, stdev= 3.82\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13760], '
'5.00th=[18816], 10.00th=[19072], '
'20.00th=[19328],\n'
' | 30.00th=[20352], '
'40.00th=[20608], '
'50.00th=[20608], '
'60.00th=[20864],\n'
' | 70.00th=[21376], '
'80.00th=[21888], '
'90.00th=[22656], '
'95.00th=[24960],\n'
' | 99.00th=[41728], '
'99.50th=[44288], '
'99.90th=[56064], '
'99.95th=[67072],\n'
' | 99.99th=[82432]\n'
' bw ( KiB/s): min=166104, '
'max=167290, per=100.00%, '
'avg=166794.15, stdev=286.54, '
'samples=13\n'
' iops : min=332210, '
'max=334580, avg=333588.46, '
'stdev=572.57, samples=13\n'
' lat (nsec) : 100=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.02%, 20=22.11%, 50=77.74%, '
'100=0.13%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=14.14%, '
'sys=83.41%, ctx=12659, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2335236,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=163MiB/s (171MB/s), '
'163MiB/s-163MiB/s '
'(171MB/s-171MB/s), io=1140MiB '
'(1196MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.output'),
'state': {'begin': 1666745268.964636,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7934114933013916,
'end': 1666745269.7580476,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.state')},
'cmd_301': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.output'),
'state': {'begin': 1666745270.7479367,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.781456470489502,
'end': 1666745271.5293932,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.state')},
'cmd_302': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.output'),
'state': {'begin': 1666745271.5315776,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.040171384811401,
'end': 1666745282.571749,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.state')},
'cmd_303': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752371,\n'
' "timestamp_ms" : '
'1666752371752,\n'
' "time" : "Wed Oct 26 04:46:11 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1196563968,\n'
' "io_kbytes" : 1168519,\n'
' "bw_bytes" : 170913293,\n'
' "bw" : 166907,\n'
' "iops" : 333813.883731,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2337031,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1152,\n'
' "max" : 24005,\n'
' "mean" : 2220.236975,\n'
' "stddev" : '
'1090.837490,\n'
' "N" : 2337032\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 3196,\n'
' "max" : 144360,\n'
' "mean" : '
'21244.316029,\n'
' "stddev" : '
'3839.831790,\n'
' "N" : 2337038,\n'
' "percentile" : {\n'
' "1.000000" : 13888,\n'
' "5.000000" : 18816,\n'
' "10.000000" : '
'19072,\n'
' "20.000000" : '
'19328,\n'
' "30.000000" : '
'20352,\n'
' "40.000000" : '
'20608,\n'
' "50.000000" : '
'20608,\n'
' "60.000000" : '
'20864,\n'
' "70.000000" : '
'21632,\n'
' "80.000000" : '
'21888,\n'
' "90.000000" : '
'22912,\n'
' "95.000000" : '
'24960,\n'
' "99.000000" : '
'41728,\n'
' "99.500000" : '
'44288,\n'
' "99.900000" : '
'53504,\n'
' "99.950000" : '
'64768,\n'
' "99.990000" : 81408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7093,\n'
' "max" : 145563,\n'
' "mean" : '
'23464.552185,\n'
' "stddev" : '
'3807.683431,\n'
' "N" : 2337038\n'
' },\n'
' "bw_min" : 166713,\n'
' "bw_max" : 167275,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'166953.357143,\n'
' "bw_dev" : 172.887517,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 333426,\n'
' "iops_max" : 334550,\n'
' "iops_mean" : '
'333906.928571,\n'
' "iops_stddev" : '
'345.829763,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.800000,\n'
' "sys_cpu" : 83.771429,\n'
' "ctx" : 12624,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999957,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.015575,\n'
' "20" : 23.539739,\n'
' "50" : 76.322522,\n'
' "100" : 0.120965,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=143058: Wed Oct 26 04:46:11 '
'2022\n'
' read: IOPS=334k, BW=163MiB/s '
'(171MB/s)(1141MiB/7001msec)\n'
' slat (nsec): min=1152, '
'max=24005, avg=2220.24, '
'stdev=1090.84\n'
' clat (usec): min=3, max=144, '
'avg=21.24, stdev= 3.84\n'
' lat (usec): min=7, max=145, '
'avg=23.46, stdev= 3.81\n'
' clat percentiles (nsec):\n'
' | 1.00th=[13888], '
'5.00th=[18816], 10.00th=[19072], '
'20.00th=[19328],\n'
' | 30.00th=[20352], '
'40.00th=[20608], '
'50.00th=[20608], '
'60.00th=[20864],\n'
' | 70.00th=[21632], '
'80.00th=[21888], '
'90.00th=[22912], '
'95.00th=[24960],\n'
' | 99.00th=[41728], '
'99.50th=[44288], '
'99.90th=[53504], '
'99.95th=[64768],\n'
' | 99.99th=[81408]\n'
' bw ( KiB/s): min=166713, '
'max=167275, per=100.00%, '
'avg=166953.36, stdev=172.89, '
'samples=14\n'
' iops : min=333426, '
'max=334550, avg=333906.93, '
'stdev=345.83, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.02%, 20=23.54%, 50=76.32%, '
'100=0.12%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=13.80%, '
'sys=83.77%, ctx=12624, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2337031,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=163MiB/s (171MB/s), '
'163MiB/s-163MiB/s '
'(171MB/s-171MB/s), io=1141MiB '
'(1197MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.output'),
'state': {'begin': 1666745282.574461,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7908134460449219,
'end': 1666745283.3652744,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.state')},
'cmd_304': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.output'),
'state': {'begin': 1666745284.3566759,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.779430627822876,
'end': 1666745285.1361065,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.state')},
'cmd_305': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.output'),
'state': {'begin': 1666745285.1381803,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring_cmd" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.040968179702759,
'end': 1666745296.1791484,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.state')},
'cmd_306': {'output': 'io_uring_cmd-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring_cmd, '
'iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752385,\n'
' "timestamp_ms" : '
'1666752385359,\n'
' "time" : "Wed Oct 26 04:46:25 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring_cmd",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1210347520,\n'
' "io_kbytes" : 1181980,\n'
' "bw_bytes" : 172882091,\n'
' "bw" : 168830,\n'
' "iops" : 337659.334381,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2363953,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1122,\n'
' "max" : 22653,\n'
' "mean" : 2034.197053,\n'
' "stddev" : '
'1011.133446,\n'
' "N" : 2363953\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 90,\n'
' "max" : 142347,\n'
' "mean" : '
'21168.681512,\n'
' "stddev" : '
'3823.819840,\n'
' "N" : 2363960,\n'
' "percentile" : {\n'
' "1.000000" : 14400,\n'
' "5.000000" : 18560,\n'
' "10.000000" : '
'18816,\n'
' "20.000000" : '
'19072,\n'
' "30.000000" : '
'20352,\n'
' "40.000000" : '
'20352,\n'
' "50.000000" : '
'20608,\n'
' "60.000000" : '
'21120,\n'
' "70.000000" : '
'21632,\n'
' "80.000000" : '
'21888,\n'
' "90.000000" : '
'22912,\n'
' "95.000000" : '
'24704,\n'
' "99.000000" : '
'41216,\n'
' "99.500000" : '
'43776,\n'
' "99.900000" : '
'55040,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 80384\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5551,\n'
' "max" : 143539,\n'
' "mean" : '
'23202.878542,\n'
' "stddev" : '
'3787.397119,\n'
' "N" : 2363960\n'
' },\n'
' "bw_min" : 168266,\n'
' "bw_max" : 169275,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'168882.153846,\n'
' "bw_dev" : 336.167579,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 336532,\n'
' "iops_max" : 338550,\n'
' "iops_mean" : '
'337764.307692,\n'
' "iops_stddev" : '
'672.444221,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.700000,\n'
' "sys_cpu" : 83.814286,\n'
' "ctx" : 12774,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999958,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 0.013790,\n'
' "20" : 23.539004,\n'
' "50" : 76.321145,\n'
' "100" : 0.124791,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-reference: '
'(groupid=0, jobs=1): err= 0: '
'pid=143096: Wed Oct 26 04:46:25 '
'2022\n'
' read: IOPS=338k, BW=165MiB/s '
'(173MB/s)(1154MiB/7001msec)\n'
' slat (nsec): min=1122, '
'max=22653, avg=2034.20, '
'stdev=1011.13\n'
' clat (nsec): min=90, '
'max=142347, avg=21168.68, '
'stdev=3823.82\n'
' lat (usec): min=5, max=143, '
'avg=23.20, stdev= 3.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14400], '
'5.00th=[18560], 10.00th=[18816], '
'20.00th=[19072],\n'
' | 30.00th=[20352], '
'40.00th=[20352], '
'50.00th=[20608], '
'60.00th=[21120],\n'
' | 70.00th=[21632], '
'80.00th=[21888], '
'90.00th=[22912], '
'95.00th=[24704],\n'
' | 99.00th=[41216], '
'99.50th=[43776], '
'99.90th=[55040], '
'99.95th=[66048],\n'
' | 99.99th=[80384]\n'
' bw ( KiB/s): min=168266, '
'max=169275, per=100.00%, '
'avg=168882.15, stdev=336.17, '
'samples=13\n'
' iops : min=336532, '
'max=338550, avg=337764.31, '
'stdev=672.44, samples=13\n'
' lat (nsec) : 100=0.01%\n'
' lat (usec) : 2=0.01%, '
'10=0.01%, 20=23.54%, 50=76.32%, '
'100=0.12%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=13.70%, '
'sys=83.81%, ctx=12774, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2363953,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=165MiB/s (173MB/s), '
'165MiB/s-165MiB/s '
'(173MB/s-173MB/s), io=1154MiB '
'(1210MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.output'),
'state': {'begin': 1666745296.182085,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7856402397155762,
'end': 1666745296.9677253,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.state')},
'cmd_307': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.output'),
'state': {'begin': 1666745297.9606037,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7840566635131836,
'end': 1666745298.7446604,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.state')},
'cmd_308': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.output'),
'state': {'begin': 1666745298.7467673,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.065226793289185,
'end': 1666745309.811994,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.state')},
'cmd_309': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752398,\n'
' "timestamp_ms" : '
'1666752398992,\n'
' "time" : "Wed Oct 26 04:46:38 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1159592448,\n'
' "io_kbytes" : 1132414,\n'
' "bw_bytes" : 165632402,\n'
' "bw" : 161750,\n'
' "iops" : 323499.642908,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2264821,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1212,\n'
' "max" : 32892,\n'
' "mean" : 2529.842174,\n'
' "stddev" : '
'737.661078,\n'
' "N" : 2264822\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 120,\n'
' "max" : 143188,\n'
' "mean" : '
'21923.724910,\n'
' "stddev" : '
'3812.563652,\n'
' "N" : 2264828,\n'
' "percentile" : {\n'
' "1.000000" : 14272,\n'
' "5.000000" : 19840,\n'
' "10.000000" : '
'20352,\n'
' "20.000000" : '
'20864,\n'
' "30.000000" : '
'21120,\n'
' "40.000000" : '
'21120,\n'
' "50.000000" : '
'21376,\n'
' "60.000000" : '
'21376,\n'
' "70.000000" : '
'21632,\n'
' "80.000000" : '
'21888,\n'
' "90.000000" : '
'22912,\n'
' "95.000000" : '
'25984,\n'
' "99.000000" : '
'42240,\n'
' "99.500000" : '
'44800,\n'
' "99.900000" : '
'55040,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 81408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6672,\n'
' "max" : 144431,\n'
' "mean" : '
'24453.567314,\n'
' "stddev" : '
'3663.939270,\n'
' "N" : 2264828\n'
' },\n'
' "bw_min" : 161462,\n'
' "bw_max" : 162202,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'161798.928571,\n'
' "bw_dev" : 238.197870,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 322924,\n'
' "iops_max" : 324404,\n'
' "iops_mean" : '
'323597.928571,\n'
' "iops_stddev" : '
'476.383889,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.985714,\n'
' "sys_cpu" : 81.714286,\n'
' "ctx" : 12576,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999978,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.018721,\n'
' "20" : 6.321383,\n'
' "50" : 93.517589,\n'
' "100" : 0.139349,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=143134: Wed '
'Oct 26 04:46:38 2022\n'
' read: IOPS=323k, BW=158MiB/s '
'(166MB/s)(1106MiB/7001msec)\n'
' slat (nsec): min=1212, '
'max=32892, avg=2529.84, '
'stdev=737.66\n'
' clat (nsec): min=120, '
'max=143188, avg=21923.72, '
'stdev=3812.56\n'
' lat (usec): min=6, max=144, '
'avg=24.45, stdev= 3.66\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14272], '
'5.00th=[19840], 10.00th=[20352], '
'20.00th=[20864],\n'
' | 30.00th=[21120], '
'40.00th=[21120], '
'50.00th=[21376], '
'60.00th=[21376],\n'
' | 70.00th=[21632], '
'80.00th=[21888], '
'90.00th=[22912], '
'95.00th=[25984],\n'
' | 99.00th=[42240], '
'99.50th=[44800], '
'99.90th=[55040], '
'99.95th=[67072],\n'
' | 99.99th=[81408]\n'
' bw ( KiB/s): min=161462, '
'max=162202, per=100.00%, '
'avg=161798.93, stdev=238.20, '
'samples=14\n'
' iops : min=322924, '
'max=324404, avg=323597.93, '
'stdev=476.38, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.02%, 20=6.32%, '
'50=93.52%\n'
' lat (usec) : 100=0.14%, '
'250=0.01%\n'
' cpu : usr=15.99%, '
'sys=81.71%, ctx=12576, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2264821,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=158MiB/s (166MB/s), '
'158MiB/s-158MiB/s '
'(166MB/s-166MB/s), io=1106MiB '
'(1160MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.output'),
'state': {'begin': 1666745309.8151252,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7887105941772461,
'end': 1666745310.6038358,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.state')},
'cmd_31': {'output': 'rm: cannot remove '
"'/tmp/fio-output_512_1_io_uring-reference_1.txt': "
'No such file or directory\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.output'),
'state': {'begin': 1666743979.7884958,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7848758697509766,
'end': 1666743980.5733716,
'err': 1,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.state')},
'cmd_310': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.output'),
'state': {'begin': 1666745311.588529,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7839469909667969,
'end': 1666745312.372476,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.state')},
'cmd_311': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.output'),
'state': {'begin': 1666745312.3744292,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.052843809127808,
'end': 1666745323.427273,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.state')},
'cmd_312': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752412,\n'
' "timestamp_ms" : '
'1666752412607,\n'
' "time" : "Wed Oct 26 04:46:52 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1175407104,\n'
' "io_kbytes" : 1147858,\n'
' "bw_bytes" : 167891316,\n'
' "bw" : 163956,\n'
' "iops" : 327911.584059,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2295709,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1202,\n'
' "max" : 46768,\n'
' "mean" : 2507.877895,\n'
' "stddev" : '
'718.916471,\n'
' "N" : 2295710\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 120,\n'
' "max" : 143369,\n'
' "mean" : '
'21623.022268,\n'
' "stddev" : '
'3822.739674,\n'
' "N" : 2295716,\n'
' "percentile" : {\n'
' "1.000000" : 14144,\n'
' "5.000000" : 19840,\n'
' "10.000000" : '
'20608,\n'
' "20.000000" : '
'20608,\n'
' "30.000000" : '
'20864,\n'
' "40.000000" : '
'20864,\n'
' "50.000000" : '
'21120,\n'
' "60.000000" : '
'21120,\n'
' "70.000000" : '
'21120,\n'
' "80.000000" : '
'21376,\n'
' "90.000000" : '
'22656,\n'
' "95.000000" : '
'25728,\n'
' "99.000000" : '
'42240,\n'
' "99.500000" : '
'44288,\n'
' "99.900000" : '
'56064,\n'
' "99.950000" : '
'68096,\n'
' "99.990000" : 83456\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6823,\n'
' "max" : 144632,\n'
' "mean" : '
'24130.900482,\n'
' "stddev" : '
'3672.577138,\n'
' "N" : 2295716\n'
' },\n'
' "bw_min" : 163612,\n'
' "bw_max" : 164316,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'164015.000000,\n'
' "bw_dev" : 228.698856,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 327224,\n'
' "iops_max" : 328633,\n'
' "iops_mean" : '
'328030.230769,\n'
' "iops_stddev" : '
'457.340893,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.928571,\n'
' "sys_cpu" : 81.728571,\n'
' "ctx" : 12743,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999978,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.016378,\n'
' "20" : 6.622529,\n'
' "50" : 93.218435,\n'
' "100" : 0.138563,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=143172: Wed '
'Oct 26 04:46:52 2022\n'
' read: IOPS=328k, BW=160MiB/s '
'(168MB/s)(1121MiB/7001msec)\n'
' slat (nsec): min=1202, '
'max=46768, avg=2507.88, '
'stdev=718.92\n'
' clat (nsec): min=120, '
'max=143369, avg=21623.02, '
'stdev=3822.74\n'
' lat (usec): min=6, max=144, '
'avg=24.13, stdev= 3.67\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14144], '
'5.00th=[19840], 10.00th=[20608], '
'20.00th=[20608],\n'
' | 30.00th=[20864], '
'40.00th=[20864], '
'50.00th=[21120], '
'60.00th=[21120],\n'
' | 70.00th=[21120], '
'80.00th=[21376], '
'90.00th=[22656], '
'95.00th=[25728],\n'
' | 99.00th=[42240], '
'99.50th=[44288], '
'99.90th=[56064], '
'99.95th=[68096],\n'
' | 99.99th=[83456]\n'
' bw ( KiB/s): min=163612, '
'max=164316, per=100.00%, '
'avg=164015.00, stdev=228.70, '
'samples=13\n'
' iops : min=327224, '
'max=328633, avg=328030.23, '
'stdev=457.34, samples=13\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.02%, 20=6.62%, '
'50=93.22%\n'
' lat (usec) : 100=0.14%, '
'250=0.01%\n'
' cpu : usr=15.93%, '
'sys=81.73%, ctx=12743, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2295709,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=160MiB/s (168MB/s), '
'160MiB/s-160MiB/s '
'(168MB/s-168MB/s), io=1121MiB '
'(1175MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.output'),
'state': {'begin': 1666745323.4303358,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7909913063049316,
'end': 1666745324.221327,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.state')},
'cmd_313': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.output'),
'state': {'begin': 1666745325.2078052,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7832224369049072,
'end': 1666745325.9910276,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.state')},
'cmd_314': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.output'),
'state': {'begin': 1666745325.9934478,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring_cmd" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/ng0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.061598300933838,
'end': 1666745337.055046,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.state')},
'cmd_315': {'output': 'io_uring_cmd-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752426,\n'
' "timestamp_ms" : '
'1666752426234,\n'
' "time" : "Wed Oct 26 04:47:06 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring_cmd",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/ng0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1161004544,\n'
' "io_kbytes" : 1133793,\n'
' "bw_bytes" : 165834101,\n'
' "bw" : 161947,\n'
' "iops" : 323893.586630,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2267579,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1202,\n'
' "max" : 38503,\n'
' "mean" : 2508.514283,\n'
' "stddev" : '
'748.390671,\n'
' "N" : 2267580\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 121,\n'
' "max" : 141366,\n'
' "mean" : '
'21924.472038,\n'
' "stddev" : '
'3826.780599,\n'
' "N" : 2267586,\n'
' "percentile" : {\n'
' "1.000000" : 14272,\n'
' "5.000000" : 19840,\n'
' "10.000000" : '
'20352,\n'
' "20.000000" : '
'20864,\n'
' "30.000000" : '
'21120,\n'
' "40.000000" : '
'21120,\n'
' "50.000000" : '
'21376,\n'
' "60.000000" : '
'21376,\n'
' "70.000000" : '
'21632,\n'
' "80.000000" : '
'22144,\n'
' "90.000000" : '
'22912,\n'
' "95.000000" : '
'25728,\n'
' "99.000000" : '
'42752,\n'
' "99.500000" : '
'44800,\n'
' "99.900000" : '
'56576,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 81408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6502,\n'
' "max" : 142618,\n'
' "mean" : '
'24432.986819,\n'
' "stddev" : '
'3677.160613,\n'
' "N" : 2267586\n'
' },\n'
' "bw_min" : 161484,\n'
' "bw_max" : 162868,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'161985.571429,\n'
' "bw_dev" : 433.250099,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 322966,\n'
' "iops_max" : 325736,\n'
' "iops_mean" : '
'323971.214286,\n'
' "iops_stddev" : '
'866.505997,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.057143,\n'
' "sys_cpu" : 80.628571,\n'
' "ctx" : 12620,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999978,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.018875,\n'
' "20" : 6.020430,\n'
' "50" : 93.815871,\n'
' "100" : 0.142884,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=143210: Wed '
'Oct 26 04:47:06 2022\n'
' read: IOPS=324k, BW=158MiB/s '
'(166MB/s)(1107MiB/7001msec)\n'
' slat (nsec): min=1202, '
'max=38503, avg=2508.51, '
'stdev=748.39\n'
' clat (nsec): min=121, '
'max=141366, avg=21924.47, '
'stdev=3826.78\n'
' lat (usec): min=6, max=142, '
'avg=24.43, stdev= 3.68\n'
' clat percentiles (nsec):\n'
' | 1.00th=[14272], '
'5.00th=[19840], 10.00th=[20352], '
'20.00th=[20864],\n'
' | 30.00th=[21120], '
'40.00th=[21120], '
'50.00th=[21376], '
'60.00th=[21376],\n'
' | 70.00th=[21632], '
'80.00th=[22144], '
'90.00th=[22912], '
'95.00th=[25728],\n'
' | 99.00th=[42752], '
'99.50th=[44800], '
'99.90th=[56576], '
'99.95th=[67072],\n'
' | 99.99th=[81408]\n'
' bw ( KiB/s): min=161484, '
'max=162868, per=100.00%, '
'avg=161985.57, stdev=433.25, '
'samples=14\n'
' iops : min=322966, '
'max=325736, avg=323971.21, '
'stdev=866.51, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.02%, 20=6.02%, '
'50=93.82%\n'
' lat (usec) : 100=0.14%, '
'250=0.01%\n'
' cpu : usr=17.06%, '
'sys=80.63%, ctx=12620, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2267579,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=158MiB/s (166MB/s), '
'158MiB/s-158MiB/s '
'(166MB/s-166MB/s), io=1107MiB '
'(1161MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.output'),
'state': {'begin': 1666745337.0583837,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.8026723861694336,
'end': 1666745337.861056,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.state')},
'cmd_316': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.output'),
'state': {'begin': 1666745339.881891,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7861189842224121,
'end': 1666745340.66801,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.state')},
'cmd_317': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.output'),
'state': {'begin': 1666745340.6700892,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.4807448387146,
'end': 1666745352.150834,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.state')},
'cmd_318': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752441,\n'
' "timestamp_ms" : '
'1666752441259,\n'
' "time" : "Wed Oct 26 04:47:21 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2276859392,\n'
' "io_kbytes" : 2223495,\n'
' "bw_bytes" : 325219167,\n'
' "bw" : 317596,\n'
' "iops" : 635192.543922,\n'
' "runtime" : 7001,\n'
' "total_ios" : 4446983,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 190,\n'
' "max" : 30808,\n'
' "mean" : 266.992913,\n'
' "stddev" : 64.891956,\n'
' "N" : 4446984\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 441,\n'
' "max" : 133430,\n'
' "mean" : '
'11928.421873,\n'
' "stddev" : '
'4431.847865,\n'
' "N" : 4446983,\n'
' "percentile" : {\n'
' "1.000000" : 7008,\n'
' "5.000000" : 7904,\n'
' "10.000000" : 8512,\n'
' "20.000000" : 9536,\n'
' "30.000000" : '
'10048,\n'
' "40.000000" : '
'10560,\n'
' "50.000000" : '
'10944,\n'
' "60.000000" : '
'11584,\n'
' "70.000000" : '
'12224,\n'
' "80.000000" : '
'13120,\n'
' "90.000000" : '
'14784,\n'
' "95.000000" : '
'19840,\n'
' "99.000000" : '
'31104,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'47872,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5991,\n'
' "max" : 133711,\n'
' "mean" : '
'12195.414785,\n'
' "stddev" : '
'4429.865740,\n'
' "N" : 4446983\n'
' },\n'
' "bw_min" : 317209,\n'
' "bw_max" : 318994,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'317838.615385,\n'
' "bw_dev" : 507.818461,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 634418,\n'
' "iops_max" : 637988,\n'
' "iops_mean" : '
'635677.538462,\n'
' "iops_stddev" : '
'1015.420079,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 24.005084,\n'
' "8" : 75.986191,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.992103,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 28.089853,\n'
' "20" : 66.951931,\n'
' "50" : 4.864332,\n'
' "100" : 0.091613,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=143259: Wed Oct 26 04:47:21 '
'2022\n'
' read: IOPS=635k, BW=310MiB/s '
'(325MB/s)(2171MiB/7001msec)\n'
' slat (nsec): min=190, '
'max=30808, avg=266.99, '
'stdev=64.89\n'
' clat (nsec): min=441, '
'max=133430, avg=11928.42, '
'stdev=4431.85\n'
' lat (usec): min=5, max=133, '
'avg=12.20, stdev= 4.43\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7008], '
'5.00th=[ 7904], 10.00th=[ 8512], '
'20.00th=[ 9536],\n'
' | 30.00th=[10048], '
'40.00th=[10560], '
'50.00th=[10944], '
'60.00th=[11584],\n'
' | 70.00th=[12224], '
'80.00th=[13120], '
'90.00th=[14784], '
'95.00th=[19840],\n'
' | 99.00th=[31104], '
'99.50th=[34560], '
'99.90th=[47872], '
'99.95th=[65280],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=317209, '
'max=318994, per=100.00%, '
'avg=317838.62, stdev=507.82, '
'samples=13\n'
' iops : min=634418, '
'max=637988, avg=635677.54, '
'stdev=1015.42, samples=13\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=28.09%, 20=66.95%, '
'50=4.86%\n'
' lat (usec) : 100=0.09%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=24.0%, 8=76.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=4446983,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=310MiB/s (325MB/s), '
'310MiB/s-310MiB/s '
'(325MB/s-325MB/s), io=2171MiB '
'(2277MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.output'),
'state': {'begin': 1666745352.1577332,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7898359298706055,
'end': 1666745352.9475691,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.state')},
'cmd_319': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.output'),
'state': {'begin': 1666745354.972949,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7823724746704102,
'end': 1666745355.7553215,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.state')},
'cmd_32': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.output'),
'state': {'begin': 1666743980.5754085,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.044062614440918,
'end': 1666743991.619471,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.state')},
'cmd_320': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.output'),
'state': {'begin': 1666745355.7576373,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.48052453994751,
'end': 1666745367.2381618,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.state')},
'cmd_321': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752456,\n'
' "timestamp_ms" : '
'1666752456347,\n'
' "time" : "Wed Oct 26 04:47:36 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1845574656,\n'
' "io_kbytes" : 1802319,\n'
' "bw_bytes" : 263615862,\n'
' "bw" : 257437,\n'
' "iops" : 514873.589487,\n'
' "runtime" : 7001,\n'
' "total_ios" : 3604630,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 170,\n'
' "max" : 9297,\n'
' "mean" : 249.728569,\n'
' "stddev" : 28.581610,\n'
' "N" : 3604631\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4809,\n'
' "max" : 130585,\n'
' "mean" : '
'14363.971523,\n'
' "stddev" : '
'5215.015737,\n'
' "N" : 3604630,\n'
' "percentile" : {\n'
' "1.000000" : 7456,\n'
' "5.000000" : 8384,\n'
' "10.000000" : 8896,\n'
' "20.000000" : 9792,\n'
' "30.000000" : '
'10560,\n'
' "40.000000" : '
'11712,\n'
' "50.000000" : '
'14144,\n'
' "60.000000" : '
'15424,\n'
' "70.000000" : '
'16512,\n'
' "80.000000" : '
'18560,\n'
' "90.000000" : '
'20096,\n'
' "95.000000" : '
'22144,\n'
' "99.000000" : '
'31616,\n'
' "99.500000" : '
'35584,\n'
' "99.900000" : '
'48384,\n'
' "99.950000" : '
'62720,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5080,\n'
' "max" : 130805,\n'
' "mean" : '
'14613.700084,\n'
' "stddev" : '
'5209.006901,\n'
' "N" : 3604630\n'
' },\n'
' "bw_min" : 256603,\n'
' "bw_max" : 259025,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'257567.500000,\n'
' "bw_dev" : 671.520634,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 513206,\n'
' "iops_max" : 518050,\n'
' "iops_mean" : '
'515135.142857,\n'
' "iops_stddev" : '
'1342.858940,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 26,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 9.715921,\n'
' "4" : 43.387421,\n'
' "8" : 46.894910,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 95.677259,\n'
' "8" : 4.322741,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 22.285450,\n'
' "20" : 67.773863,\n'
' "50" : 9.846531,\n'
' "100" : 0.092104,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=143310: Wed Oct 26 04:47:36 '
'2022\n'
' read: IOPS=515k, BW=251MiB/s '
'(264MB/s)(1760MiB/7001msec)\n'
' slat (nsec): min=170, '
'max=9297, avg=249.73, '
'stdev=28.58\n'
' clat (usec): min=4, max=130, '
'avg=14.36, stdev= 5.22\n'
' lat (usec): min=5, max=130, '
'avg=14.61, stdev= 5.21\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7456], '
'5.00th=[ 8384], 10.00th=[ 8896], '
'20.00th=[ 9792],\n'
' | 30.00th=[10560], '
'40.00th=[11712], '
'50.00th=[14144], '
'60.00th=[15424],\n'
' | 70.00th=[16512], '
'80.00th=[18560], '
'90.00th=[20096], '
'95.00th=[22144],\n'
' | 99.00th=[31616], '
'99.50th=[35584], '
'99.90th=[48384], '
'99.95th=[62720],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=256603, '
'max=259025, per=100.00%, '
'avg=257567.50, stdev=671.52, '
'samples=14\n'
' iops : min=513206, '
'max=518050, avg=515135.14, '
'stdev=1342.86, samples=14\n'
' lat (usec) : 10=22.29%, '
'20=67.77%, 50=9.85%, 100=0.09%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=26, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=9.7%, '
'4=43.4%, 8=46.9%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=95.7%, 8=4.3%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=3604630,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=251MiB/s (264MB/s), '
'251MiB/s-251MiB/s '
'(264MB/s-264MB/s), io=1760MiB '
'(1846MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.output'),
'state': {'begin': 1666745367.241164,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.819014310836792,
'end': 1666745368.0601783,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.state')},
'cmd_322': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.output'),
'state': {'begin': 1666745370.0823493,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7883975505828857,
'end': 1666745370.8707469,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.state')},
'cmd_323': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.output'),
'state': {'begin': 1666745370.872893,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring_cmd-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="0" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.4997878074646,
'end': 1666745382.372681,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.state')},
'cmd_324': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752471,\n'
' "timestamp_ms" : '
'1666752471482,\n'
' "time" : "Wed Oct 26 04:47:51 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring_cmd-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "0",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'2261216256,\n'
' "io_kbytes" : 2208219,\n'
' "bw_bytes" : 322984753,\n'
' "bw" : 315414,\n'
' "iops" : 630828.595915,\n'
' "runtime" : 7001,\n'
' "total_ios" : 4416431,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 190,\n'
' "max" : 24937,\n'
' "mean" : 265.485439,\n'
' "stddev" : 66.086546,\n'
' "N" : 4416431\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 921,\n'
' "max" : 131396,\n'
' "mean" : '
'12004.528837,\n'
' "stddev" : '
'4455.386441,\n'
' "N" : 4416438,\n'
' "percentile" : {\n'
' "1.000000" : 6816,\n'
' "5.000000" : 7904,\n'
' "10.000000" : 8512,\n'
' "20.000000" : 9536,\n'
' "30.000000" : '
'10176,\n'
' "40.000000" : '
'10688,\n'
' "50.000000" : '
'11072,\n'
' "60.000000" : '
'11712,\n'
' "70.000000" : '
'12480,\n'
' "80.000000" : '
'13248,\n'
' "90.000000" : '
'14784,\n'
' "95.000000" : '
'19840,\n'
' "99.000000" : '
'31104,\n'
' "99.500000" : '
'34560,\n'
' "99.900000" : '
'49920,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5580,\n'
' "max" : 131617,\n'
' "mean" : '
'12270.014258,\n'
' "stddev" : '
'4453.681752,\n'
' "N" : 4416438\n'
' },\n'
' "bw_min" : 314606,\n'
' "bw_max" : 317057,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'315565.928571,\n'
' "bw_dev" : 753.328967,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 629212,\n'
' "iops_max" : 634114,\n'
' "iops_mean" : '
'631131.928571,\n'
' "iops_stddev" : '
'1506.587610,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 99.985714,\n'
' "sys_cpu" : 0.000000,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.100000,\n'
' "2" : 0.100000,\n'
' "4" : 25.115280,\n'
' "8" : 74.875663,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.991752,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 28.039406,\n'
' "20" : 66.984359,\n'
' "50" : 4.876879,\n'
' "100" : 0.097998,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring_cmd-bdev_xnvme: '
'(groupid=0, jobs=1): err= 0: '
'pid=143360: Wed Oct 26 04:47:51 '
'2022\n'
' read: IOPS=631k, BW=308MiB/s '
'(323MB/s)(2156MiB/7001msec)\n'
' slat (nsec): min=190, '
'max=24937, avg=265.49, '
'stdev=66.09\n'
' clat (nsec): min=921, '
'max=131396, avg=12004.53, '
'stdev=4455.39\n'
' lat (usec): min=5, max=131, '
'avg=12.27, stdev= 4.45\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 6816], '
'5.00th=[ 7904], 10.00th=[ 8512], '
'20.00th=[ 9536],\n'
' | 30.00th=[10176], '
'40.00th=[10688], '
'50.00th=[11072], '
'60.00th=[11712],\n'
' | 70.00th=[12480], '
'80.00th=[13248], '
'90.00th=[14784], '
'95.00th=[19840],\n'
' | 99.00th=[31104], '
'99.50th=[34560], '
'99.90th=[49920], '
'99.95th=[67072],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=314606, '
'max=317057, per=100.00%, '
'avg=315565.93, stdev=753.33, '
'samples=14\n'
' iops : min=629212, '
'max=634114, avg=631131.93, '
'stdev=1506.59, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=28.04%, 20=66.98%, '
'50=4.88%\n'
' lat (usec) : 100=0.10%, '
'250=0.01%\n'
' cpu : usr=99.99%, '
'sys=0.00%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=0.1%, 2=0.1%, '
'4=25.1%, 8=74.9%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=4416431,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=308MiB/s (323MB/s), '
'308MiB/s-308MiB/s '
'(323MB/s-323MB/s), io=2156MiB '
'(2261MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.output'),
'state': {'begin': 1666745382.375356,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.828303337097168,
'end': 1666745383.2036593,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.state')},
'cmd_325': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.output'),
'state': {'begin': 1666745384.217071,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.779996395111084,
'end': 1666745384.9970675,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.state')},
'cmd_326': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.output'),
'state': {'begin': 1666745384.99917,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.040830373764038,
'end': 1666745396.0400004,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.state')},
'cmd_327': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752485,\n'
' "timestamp_ms" : '
'1666752485218,\n'
' "time" : "Wed Oct 26 04:48:05 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1124852224,\n'
' "io_kbytes" : 1098488,\n'
' "bw_bytes" : 160670221,\n'
' "bw" : 156904,\n'
' "iops" : 313807.884588,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2196969,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1282,\n'
' "max" : 42510,\n'
' "mean" : 2472.162358,\n'
' "stddev" : '
'910.605618,\n'
' "N" : 2196970\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 152536,\n'
' "mean" : '
'22632.731950,\n'
' "stddev" : '
'3858.580981,\n'
' "N" : 2196976,\n'
' "percentile" : {\n'
' "1.000000" : 15552,\n'
' "5.000000" : 20352,\n'
' "10.000000" : '
'20352,\n'
' "20.000000" : '
'20864,\n'
' "30.000000" : '
'21632,\n'
' "40.000000" : '
'21888,\n'
' "50.000000" : '
'22144,\n'
' "60.000000" : '
'22400,\n'
' "70.000000" : '
'23168,\n'
' "80.000000" : '
'23424,\n'
' "90.000000" : '
'23936,\n'
' "95.000000" : '
'26496,\n'
' "99.000000" : '
'42752,\n'
' "99.500000" : '
'45312,\n'
' "99.900000" : '
'54016,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 81408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6793,\n'
' "max" : 153869,\n'
' "mean" : '
'25104.895035,\n'
' "stddev" : '
'3768.630555,\n'
' "N" : 2196976\n'
' },\n'
' "bw_min" : 156481,\n'
' "bw_max" : 157772,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'156978.923077,\n'
' "bw_dev" : 443.567819,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 312962,\n'
' "iops_max" : 315544,\n'
' "iops_mean" : '
'313957.923077,\n'
' "iops_stddev" : '
'887.118036,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.042857,\n'
' "sys_cpu" : 82.785714,\n'
' "ctx" : 12432,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999954,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.022486,\n'
' "20" : 3.535780,\n'
' "50" : 96.305455,\n'
' "100" : 0.129815,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 3157749,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 17441,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 17441,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=143399: Wed '
'Oct 26 04:48:05 2022\n'
' read: IOPS=314k, BW=153MiB/s '
'(161MB/s)(1073MiB/7001msec)\n'
' slat (nsec): min=1282, '
'max=42510, avg=2472.16, '
'stdev=910.61\n'
' clat (nsec): min=80, '
'max=152536, avg=22632.73, '
'stdev=3858.58\n'
' lat (usec): min=6, max=153, '
'avg=25.10, stdev= 3.77\n'
' clat percentiles (nsec):\n'
' | 1.00th=[15552], '
'5.00th=[20352], 10.00th=[20352], '
'20.00th=[20864],\n'
' | 30.00th=[21632], '
'40.00th=[21888], '
'50.00th=[22144], '
'60.00th=[22400],\n'
' | 70.00th=[23168], '
'80.00th=[23424], '
'90.00th=[23936], '
'95.00th=[26496],\n'
' | 99.00th=[42752], '
'99.50th=[45312], '
'99.90th=[54016], '
'99.95th=[66048],\n'
' | 99.99th=[81408]\n'
' bw ( KiB/s): min=156481, '
'max=157772, per=100.00%, '
'avg=156978.92, stdev=443.57, '
'samples=13\n'
' iops : min=312962, '
'max=315544, avg=313957.92, '
'stdev=887.12, samples=13\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.02%, 20=3.54%, '
'50=96.31%\n'
' lat (usec) : 100=0.13%, '
'250=0.01%\n'
' cpu : usr=15.04%, '
'sys=82.79%, ctx=12432, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2196969,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=153MiB/s (161MB/s), '
'153MiB/s-153MiB/s '
'(161MB/s-161MB/s), io=1073MiB '
'(1125MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=3157749/0, '
'merge=0/0, ticks=17441/0, '
'in_queue=17441, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.output'),
'state': {'begin': 1666745396.042881,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8225052356719971,
'end': 1666745396.8653862,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.state')},
'cmd_328': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.output'),
'state': {'begin': 1666745397.8776748,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7909982204437256,
'end': 1666745398.668673,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.state')},
'cmd_329': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.output'),
'state': {'begin': 1666745398.6709023,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.03939437866211,
'end': 1666745409.7102966,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.state')},
'cmd_33': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751080,\n'
' "timestamp_ms" : '
'1666751080821,\n'
' "time" : "Wed Oct 26 04:24:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 284537856,\n'
' "io_kbytes" : 277869,\n'
' "bw_bytes" : 40642459,\n'
' "bw" : 39689,\n'
' "iops" : 79379.802885,\n'
' "runtime" : 7001,\n'
' "total_ios" : 555738,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2424,\n'
' "max" : 32170,\n'
' "mean" : 2650.417359,\n'
' "stddev" : 227.123682,\n'
' "N" : 555738\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 161,\n'
' "max" : 84729,\n'
' "mean" : 9278.076117,\n'
' "stddev" : '
'1249.290259,\n'
' "N" : 555738,\n'
' "percentile" : {\n'
' "1.000000" : 8896,\n'
' "5.000000" : 8896,\n'
' "10.000000" : 9024,\n'
' "20.000000" : 9024,\n'
' "30.000000" : 9024,\n'
' "40.000000" : 9024,\n'
' "50.000000" : 9152,\n'
' "60.000000" : 9152,\n'
' "70.000000" : 9152,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9280,\n'
' "95.000000" : 9536,\n'
' "99.000000" : 15296,\n'
' "99.500000" : 15936,\n'
' "99.900000" : 19328,\n'
' "99.950000" : 27264,\n'
' "99.990000" : 55552\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8526,\n'
' "max" : 87364,\n'
' "mean" : 11928.493475,\n'
' "stddev" : '
'1269.367601,\n'
' "N" : 555738\n'
' },\n'
' "bw_min" : 39567,\n'
' "bw_max" : 39849,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'39706.571429,\n'
' "bw_dev" : 93.082027,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 79134,\n'
' "iops_max" : 79698,\n'
' "iops_mean" : '
'79413.142857,\n'
' "iops_stddev" : '
'186.164054,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.571429,\n'
' "sys_cpu" : 49.257143,\n'
' "ctx" : 555625,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.871188,\n'
' "20" : 3.046580,\n'
' "50" : 0.057581,\n'
' "100" : 0.016555,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 797598,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 4985,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 4985,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=139097: Wed '
'Oct 26 04:24:40 2022\n'
' read: IOPS=79.4k, BW=38.8MiB/s '
'(40.6MB/s)(271MiB/7001msec)\n'
' slat (nsec): min=2424, '
'max=32170, avg=2650.42, '
'stdev=227.12\n'
' clat (nsec): min=161, '
'max=84729, avg=9278.08, '
'stdev=1249.29\n'
' lat (nsec): min=8526, '
'max=87364, avg=11928.49, '
'stdev=1269.37\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8896], 5.00th=[ '
'8896], 10.00th=[ 9024], 20.00th=[ '
'9024],\n'
' | 30.00th=[ 9024], 40.00th=[ '
'9024], 50.00th=[ 9152], 60.00th=[ '
'9152],\n'
' | 70.00th=[ 9152], 80.00th=[ '
'9152], 90.00th=[ 9280], 95.00th=[ '
'9536],\n'
' | 99.00th=[15296], '
'99.50th=[15936], 99.90th=[19328], '
'99.95th=[27264],\n'
' | 99.99th=[55552]\n'
' bw ( KiB/s): min=39567, '
'max=39849, per=100.00%, '
'avg=39706.57, stdev=93.08, '
'samples=14\n'
' iops : min=79134, '
'max=79698, avg=79413.14, '
'stdev=186.16, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 10=96.87%, '
'20=3.05%, 50=0.06%, 100=0.02%\n'
' cpu : usr=11.57%, '
'sys=49.26%, ctx=555625, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=555738,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=38.8MiB/s (40.6MB/s), '
'38.8MiB/s-38.8MiB/s '
'(40.6MB/s-40.6MB/s), io=271MiB '
'(285MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=797598/0, '
'merge=0/0, ticks=4985/0, '
'in_queue=4985, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.output'),
'state': {'begin': 1666743991.6221285,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.8044507503509521,
'end': 1666743992.4265792,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.state')},
'cmd_330': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752498,\n'
' "timestamp_ms" : '
'1666752498888,\n'
' "time" : "Wed Oct 26 04:48:18 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1118396416,\n'
' "io_kbytes" : 1092184,\n'
' "bw_bytes" : 159748095,\n'
' "bw" : 156003,\n'
' "iops" : 312006.999000,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2184361,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1282,\n'
' "max" : 29055,\n'
' "mean" : 2484.140099,\n'
' "stddev" : '
'984.021916,\n'
' "N" : 2184361\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 70,\n'
' "max" : 126137,\n'
' "mean" : '
'22762.563038,\n'
' "stddev" : '
'3843.773518,\n'
' "N" : 2184368,\n'
' "percentile" : {\n'
' "1.000000" : 15936,\n'
' "5.000000" : 20352,\n'
' "10.000000" : '
'20608,\n'
' "20.000000" : '
'20864,\n'
' "30.000000" : '
'21632,\n'
' "40.000000" : '
'21888,\n'
' "50.000000" : '
'22144,\n'
' "60.000000" : '
'22656,\n'
' "70.000000" : '
'23424,\n'
' "80.000000" : '
'23680,\n'
' "90.000000" : '
'23936,\n'
' "95.000000" : '
'26752,\n'
' "99.000000" : '
'42752,\n'
' "99.500000" : '
'45312,\n'
' "99.900000" : '
'54528,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 80384\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6782,\n'
' "max" : 127519,\n'
' "mean" : '
'25246.704148,\n'
' "stddev" : '
'3767.691495,\n'
' "N" : 2184368\n'
' },\n'
' "bw_min" : 155626,\n'
' "bw_max" : 156619,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'156072.692308,\n'
' "bw_dev" : 295.570461,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 311252,\n'
' "iops_max" : 313238,\n'
' "iops_mean" : '
'312145.692308,\n'
' "iops_stddev" : '
'591.374160,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.014286,\n'
' "sys_cpu" : 83.885714,\n'
' "ctx" : 12410,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999954,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.028109,\n'
' "20" : 3.435879,\n'
' "50" : 96.395696,\n'
' "100" : 0.135875,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 3135878,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 17469,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 17469,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=143437: Wed '
'Oct 26 04:48:18 2022\n'
' read: IOPS=312k, BW=152MiB/s '
'(160MB/s)(1067MiB/7001msec)\n'
' slat (nsec): min=1282, '
'max=29055, avg=2484.14, '
'stdev=984.02\n'
' clat (nsec): min=70, '
'max=126137, avg=22762.56, '
'stdev=3843.77\n'
' lat (usec): min=6, max=127, '
'avg=25.25, stdev= 3.77\n'
' clat percentiles (nsec):\n'
' | 1.00th=[15936], '
'5.00th=[20352], 10.00th=[20608], '
'20.00th=[20864],\n'
' | 30.00th=[21632], '
'40.00th=[21888], '
'50.00th=[22144], '
'60.00th=[22656],\n'
' | 70.00th=[23424], '
'80.00th=[23680], '
'90.00th=[23936], '
'95.00th=[26752],\n'
' | 99.00th=[42752], '
'99.50th=[45312], '
'99.90th=[54528], '
'99.95th=[66048],\n'
' | 99.99th=[80384]\n'
' bw ( KiB/s): min=155626, '
'max=156619, per=100.00%, '
'avg=156072.69, stdev=295.57, '
'samples=13\n'
' iops : min=311252, '
'max=313238, avg=312145.69, '
'stdev=591.37, samples=13\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.03%, 20=3.44%, '
'50=96.40%\n'
' lat (usec) : 100=0.14%, '
'250=0.01%\n'
' cpu : usr=14.01%, '
'sys=83.89%, ctx=12410, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2184361,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=152MiB/s (160MB/s), '
'152MiB/s-152MiB/s '
'(160MB/s-160MB/s), io=1067MiB '
'(1118MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=3135878/0, '
'merge=0/0, ticks=17469/0, '
'in_queue=17469, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.output'),
'state': {'begin': 1666745409.7132373,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7861456871032715,
'end': 1666745410.499383,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.state')},
'cmd_331': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.output'),
'state': {'begin': 1666745411.4932039,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7851612567901611,
'end': 1666745412.2783651,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.state')},
'cmd_332': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.output'),
'state': {'begin': 1666745412.280575,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.046215057373047,
'end': 1666745423.32679,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.state')},
'cmd_333': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752512,\n'
' "timestamp_ms" : '
'1666752512505,\n'
' "time" : "Wed Oct 26 04:48:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1115583488,\n'
' "io_kbytes" : 1089437,\n'
' "bw_bytes" : 159346305,\n'
' "bw" : 155611,\n'
' "iops" : 311222.111127,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2178866,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1292,\n'
' "max" : 72896,\n'
' "mean" : 2499.882950,\n'
' "stddev" : '
'1010.452035,\n'
' "N" : 2178867\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 80,\n'
' "max" : 140895,\n'
' "mean" : '
'22828.623283,\n'
' "stddev" : '
'3883.009481,\n'
' "N" : 2178873,\n'
' "percentile" : {\n'
' "1.000000" : 15936,\n'
' "5.000000" : 20352,\n'
' "10.000000" : '
'20608,\n'
' "20.000000" : '
'20864,\n'
' "30.000000" : '
'21632,\n'
' "40.000000" : '
'21888,\n'
' "50.000000" : '
'22400,\n'
' "60.000000" : '
'23168,\n'
' "70.000000" : '
'23424,\n'
' "80.000000" : '
'23680,\n'
' "90.000000" : '
'23936,\n'
' "95.000000" : '
'26752,\n'
' "99.000000" : '
'42752,\n'
' "99.500000" : '
'45312,\n'
' "99.900000" : '
'57600,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 81408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6842,\n'
' "max" : 142227,\n'
' "mean" : '
'25328.506996,\n'
' "stddev" : '
'3810.015386,\n'
' "N" : 2178873\n'
' },\n'
' "bw_min" : 155507,\n'
' "bw_max" : 155865,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'155698.769231,\n'
' "bw_dev" : 108.354475,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 311014,\n'
' "iops_max" : 311730,\n'
' "iops_mean" : '
'311397.538462,\n'
' "iops_stddev" : '
'216.708951,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.385714,\n'
' "sys_cpu" : 80.500000,\n'
' "ctx" : 12497,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999954,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.010000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.027262,\n'
' "20" : 3.408792,\n'
' "50" : 96.409784,\n'
' "100" : 0.148013,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 3131246,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 17492,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 17492,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=143476: Wed '
'Oct 26 04:48:32 2022\n'
' read: IOPS=311k, BW=152MiB/s '
'(159MB/s)(1064MiB/7001msec)\n'
' slat (nsec): min=1292, '
'max=72896, avg=2499.88, '
'stdev=1010.45\n'
' clat (nsec): min=80, '
'max=140895, avg=22828.62, '
'stdev=3883.01\n'
' lat (usec): min=6, max=142, '
'avg=25.33, stdev= 3.81\n'
' clat percentiles (nsec):\n'
' | 1.00th=[15936], '
'5.00th=[20352], 10.00th=[20608], '
'20.00th=[20864],\n'
' | 30.00th=[21632], '
'40.00th=[21888], '
'50.00th=[22400], '
'60.00th=[23168],\n'
' | 70.00th=[23424], '
'80.00th=[23680], '
'90.00th=[23936], '
'95.00th=[26752],\n'
' | 99.00th=[42752], '
'99.50th=[45312], '
'99.90th=[57600], '
'99.95th=[67072],\n'
' | 99.99th=[81408]\n'
' bw ( KiB/s): min=155507, '
'max=155865, per=100.00%, '
'avg=155698.77, stdev=108.35, '
'samples=13\n'
' iops : min=311014, '
'max=311730, avg=311397.54, '
'stdev=216.71, samples=13\n'
' lat (nsec) : 100=0.01%, '
'250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.03%, 20=3.41%, '
'50=96.41%\n'
' lat (usec) : 100=0.15%, '
'250=0.01%\n'
' cpu : usr=17.39%, '
'sys=80.50%, ctx=12497, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2178866,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=152MiB/s (159MB/s), '
'152MiB/s-152MiB/s '
'(159MB/s-159MB/s), io=1064MiB '
'(1116MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=3131246/0, '
'merge=0/0, ticks=17492/0, '
'in_queue=17492, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.output'),
'state': {'begin': 1666745423.3296907,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.78910231590271,
'end': 1666745424.118793,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.state')},
'cmd_334': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.output'),
'state': {'begin': 1666745426.137515,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.786327600479126,
'end': 1666745426.9238427,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.state')},
'cmd_335': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.output'),
'state': {'begin': 1666745426.9262161,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.461182594299316,
'end': 1666745438.3873987,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.state')},
'cmd_336': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752527,\n'
' "timestamp_ms" : '
'1666752527493,\n'
' "time" : "Wed Oct 26 04:48:47 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1236315648,\n'
' "io_kbytes" : 1207339,\n'
' "bw_bytes" : 176591293,\n'
' "bw" : 172452,\n'
' "iops" : 344903.870876,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2414672,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 120,\n'
' "max" : 12142,\n'
' "mean" : 141.289363,\n'
' "stddev" : 45.104120,\n'
' "N" : 2414672\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 19116,\n'
' "max" : 118853,\n'
' "mean" : '
'21467.040025,\n'
' "stddev" : '
'2785.766655,\n'
' "N" : 2414672,\n'
' "percentile" : {\n'
' "1.000000" : 19584,\n'
' "5.000000" : 19840,\n'
' "10.000000" : '
'20096,\n'
' "20.000000" : '
'20096,\n'
' "30.000000" : '
'20352,\n'
' "40.000000" : '
'20608,\n'
' "50.000000" : '
'20864,\n'
' "60.000000" : '
'21120,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21632,\n'
' "90.000000" : '
'21888,\n'
' "95.000000" : '
'27520,\n'
' "99.000000" : '
'33536,\n'
' "99.500000" : '
'37632,\n'
' "99.900000" : '
'49920,\n'
' "99.950000" : '
'60672,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 19247,\n'
' "max" : 119083,\n'
' "mean" : '
'21608.329388,\n'
' "stddev" : '
'2789.818426,\n'
' "N" : 2414672\n'
' },\n'
' "bw_min" : 172136,\n'
' "bw_max" : 172888,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'172497.071429,\n'
' "bw_dev" : 232.881767,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 344272,\n'
' "iops_max" : 345776,\n'
' "iops_mean" : '
'344994.142857,\n'
' "iops_stddev" : '
'465.763535,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 43.014286,\n'
' "sys_cpu" : 56.957143,\n'
' "ctx" : 21,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 12.500000,\n'
' "2" : 25.000000,\n'
' "4" : 50.000000,\n'
' "8" : 12.500000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 88.888889,\n'
' "8" : 11.111111,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 11.204420,\n'
' "50" : 88.695897,\n'
' "100" : 0.098978,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=143526: Wed '
'Oct 26 04:48:47 2022\n'
' read: IOPS=345k, BW=168MiB/s '
'(177MB/s)(1179MiB/7001msec)\n'
' slat (nsec): min=120, '
'max=12142, avg=141.29, '
'stdev=45.10\n'
' clat (usec): min=19, '
'max=118, avg=21.47, stdev= 2.79\n'
' lat (usec): min=19, '
'max=119, avg=21.61, stdev= 2.79\n'
' clat percentiles (nsec):\n'
' | 1.00th=[19584], '
'5.00th=[19840], 10.00th=[20096], '
'20.00th=[20096],\n'
' | 30.00th=[20352], '
'40.00th=[20608], '
'50.00th=[20864], '
'60.00th=[21120],\n'
' | 70.00th=[21376], '
'80.00th=[21632], '
'90.00th=[21888], '
'95.00th=[27520],\n'
' | 99.00th=[33536], '
'99.50th=[37632], '
'99.90th=[49920], '
'99.95th=[60672],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=172136, '
'max=172888, per=100.00%, '
'avg=172497.07, stdev=232.88, '
'samples=14\n'
' iops : min=344272, '
'max=345776, avg=344994.14, '
'stdev=465.76, samples=14\n'
' lat (usec) : 20=11.20%, '
'50=88.70%, 100=0.10%, 250=0.01%\n'
' cpu : usr=43.01%, '
'sys=56.96%, ctx=21, majf=0, '
'minf=0\n'
' IO depths : 1=12.5%, '
'2=25.0%, 4=50.0%, 8=12.5%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=88.9%, 8=11.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2414672,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=168MiB/s (177MB/s), '
'168MiB/s-168MiB/s '
'(177MB/s-177MB/s), io=1179MiB '
'(1236MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.output'),
'state': {'begin': 1666745438.390226,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.804879903793335,
'end': 1666745439.1951058,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.state')},
'cmd_337': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.output'),
'state': {'begin': 1666745441.2478127,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7863655090332031,
'end': 1666745442.0341783,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.state')},
'cmd_338': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.output'),
'state': {'begin': 1666745442.0363488,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.445159196853638,
'end': 1666745453.481508,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.state')},
'cmd_339': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752542,\n'
' "timestamp_ms" : '
'1666752542587,\n'
' "time" : "Wed Oct 26 04:49:02 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1230662144,\n'
' "io_kbytes" : 1201818,\n'
' "bw_bytes" : 175783765,\n'
' "bw" : 171663,\n'
' "iops" : 343326.667619,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2403630,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 24797,\n'
' "mean" : 132.649914,\n'
' "stddev" : 44.296597,\n'
' "N" : 2403630\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 16370,\n'
' "max" : 105507,\n'
' "mean" : '
'21585.856196,\n'
' "stddev" : '
'2810.338268,\n'
' "N" : 2403630,\n'
' "percentile" : {\n'
' "1.000000" : 19584,\n'
' "5.000000" : 19840,\n'
' "10.000000" : '
'20096,\n'
' "20.000000" : '
'20352,\n'
' "30.000000" : '
'20608,\n'
' "40.000000" : '
'20864,\n'
' "50.000000" : '
'21120,\n'
' "60.000000" : '
'21120,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21632,\n'
' "90.000000" : '
'22144,\n'
' "95.000000" : '
'27008,\n'
' "99.000000" : '
'35584,\n'
' "99.500000" : '
'38144,\n'
' "99.900000" : '
'51456,\n'
' "99.950000" : '
'61184,\n'
' "99.990000" : 71168\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 16490,\n'
' "max" : 105708,\n'
' "mean" : '
'21718.506110,\n'
' "stddev" : '
'2814.226828,\n'
' "N" : 2403630\n'
' },\n'
' "bw_min" : 171328,\n'
' "bw_max" : 172104,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'171710.785714,\n'
' "bw_dev" : 208.705231,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 342656,\n'
' "iops_max" : 344208,\n'
' "iops_mean" : '
'343421.571429,\n'
' "iops_stddev" : '
'417.410463,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.128571,\n'
' "sys_cpu" : 60.842857,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 12.500010,\n'
' "2" : 25.000021,\n'
' "4" : 50.000000,\n'
' "8" : 12.499969,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 88.888881,\n'
' "8" : 11.111119,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 7.242213,\n'
' "50" : 92.644500,\n'
' "100" : 0.112455,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=143575: Wed '
'Oct 26 04:49:02 2022\n'
' read: IOPS=343k, BW=168MiB/s '
'(176MB/s)(1174MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=24797, avg=132.65, '
'stdev=44.30\n'
' clat (usec): min=16, '
'max=105, avg=21.59, stdev= 2.81\n'
' lat (usec): min=16, '
'max=105, avg=21.72, stdev= 2.81\n'
' clat percentiles (nsec):\n'
' | 1.00th=[19584], '
'5.00th=[19840], 10.00th=[20096], '
'20.00th=[20352],\n'
' | 30.00th=[20608], '
'40.00th=[20864], '
'50.00th=[21120], '
'60.00th=[21120],\n'
' | 70.00th=[21376], '
'80.00th=[21632], '
'90.00th=[22144], '
'95.00th=[27008],\n'
' | 99.00th=[35584], '
'99.50th=[38144], '
'99.90th=[51456], '
'99.95th=[61184],\n'
' | 99.99th=[71168]\n'
' bw ( KiB/s): min=171328, '
'max=172104, per=100.00%, '
'avg=171710.79, stdev=208.71, '
'samples=14\n'
' iops : min=342656, '
'max=344208, avg=343421.57, '
'stdev=417.41, samples=14\n'
' lat (usec) : 20=7.24%, '
'50=92.64%, 100=0.11%, 250=0.01%\n'
' cpu : usr=39.13%, '
'sys=60.84%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=12.5%, '
'2=25.0%, 4=50.0%, 8=12.5%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=88.9%, 8=11.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2403630,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=168MiB/s (176MB/s), '
'168MiB/s-168MiB/s '
'(176MB/s-176MB/s), io=1174MiB '
'(1231MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.output'),
'state': {'begin': 1666745453.484458,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7894268035888672,
'end': 1666745454.2738848,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.state')},
'cmd_34': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.output'),
'state': {'begin': 1666743993.4221244,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7811994552612305,
'end': 1666743994.2033238,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.state')},
'cmd_340': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.output'),
'state': {'begin': 1666745456.3527138,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7916009426116943,
'end': 1666745457.1443148,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.state')},
'cmd_341': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.output'),
'state': {'begin': 1666745457.1465938,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.462625741958618,
'end': 1666745468.6092196,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.state')},
'cmd_342': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752557,\n'
' "timestamp_ms" : '
'1666752557715,\n'
' "time" : "Wed Oct 26 04:49:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1238494720,\n'
' "io_kbytes" : 1209467,\n'
' "bw_bytes" : 176902545,\n'
' "bw" : 172756,\n'
' "iops" : 345511.784031,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2418928,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 110,\n'
' "max" : 11802,\n'
' "mean" : 126.931602,\n'
' "stddev" : 38.723132,\n'
' "N" : 2418928\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 19076,\n'
' "max" : 118472,\n'
' "mean" : '
'21508.297008,\n'
' "stddev" : '
'2755.141185,\n'
' "N" : 2418928,\n'
' "percentile" : {\n'
' "1.000000" : 19584,\n'
' "5.000000" : 19840,\n'
' "10.000000" : '
'20096,\n'
' "20.000000" : '
'20352,\n'
' "30.000000" : '
'20608,\n'
' "40.000000" : '
'20864,\n'
' "50.000000" : '
'20864,\n'
' "60.000000" : '
'21120,\n'
' "70.000000" : '
'21376,\n'
' "80.000000" : '
'21632,\n'
' "90.000000" : '
'22144,\n'
' "95.000000" : '
'27008,\n'
' "99.000000" : '
'33536,\n'
' "99.500000" : '
'37632,\n'
' "99.900000" : '
'50432,\n'
' "99.950000" : '
'60672,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 19196,\n'
' "max" : 118672,\n'
' "mean" : '
'21635.228610,\n'
' "stddev" : '
'2758.482082,\n'
' "N" : 2418928\n'
' },\n'
' "bw_min" : 172512,\n'
' "bw_max" : 173032,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'172796.571429,\n'
' "bw_dev" : 170.218735,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 345024,\n'
' "iops_max" : 346064,\n'
' "iops_mean" : '
'345593.214286,\n'
' "iops_stddev" : '
'340.424662,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 44.100000,\n'
' "sys_cpu" : 55.871429,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 12.500000,\n'
' "2" : 25.000000,\n'
' "4" : 50.000000,\n'
' "8" : 12.500000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 88.888889,\n'
' "8" : 11.111111,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 7.970183,\n'
' "50" : 91.925431,\n'
' "100" : 0.102401,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=143624: Wed '
'Oct 26 04:49:17 2022\n'
' read: IOPS=346k, BW=169MiB/s '
'(177MB/s)(1181MiB/7001msec)\n'
' slat (nsec): min=110, '
'max=11802, avg=126.93, '
'stdev=38.72\n'
' clat (usec): min=19, '
'max=118, avg=21.51, stdev= 2.76\n'
' lat (usec): min=19, '
'max=118, avg=21.64, stdev= 2.76\n'
' clat percentiles (nsec):\n'
' | 1.00th=[19584], '
'5.00th=[19840], 10.00th=[20096], '
'20.00th=[20352],\n'
' | 30.00th=[20608], '
'40.00th=[20864], '
'50.00th=[20864], '
'60.00th=[21120],\n'
' | 70.00th=[21376], '
'80.00th=[21632], '
'90.00th=[22144], '
'95.00th=[27008],\n'
' | 99.00th=[33536], '
'99.50th=[37632], '
'99.90th=[50432], '
'99.95th=[60672],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=172512, '
'max=173032, per=100.00%, '
'avg=172796.57, stdev=170.22, '
'samples=14\n'
' iops : min=345024, '
'max=346064, avg=345593.21, '
'stdev=340.42, samples=14\n'
' lat (usec) : 20=7.97%, '
'50=91.93%, 100=0.10%, 250=0.01%\n'
' cpu : usr=44.10%, '
'sys=55.87%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=12.5%, '
'2=25.0%, 4=50.0%, 8=12.5%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=88.9%, 8=11.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2418928,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=169MiB/s (177MB/s), '
'169MiB/s-169MiB/s '
'(177MB/s-177MB/s), io=1181MiB '
'(1238MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.output'),
'state': {'begin': 1666745468.6123612,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.795048713684082,
'end': 1666745469.40741,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.state')},
'cmd_343': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.output'),
'state': {'begin': 1666745471.5778756,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7822365760803223,
'end': 1666745472.3601122,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.state')},
'cmd_344': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.output'),
'state': {'begin': 1666745472.3624802,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.46537733078003,
'end': 1666745483.8278575,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.state')},
'cmd_345': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752572,\n'
' "timestamp_ms" : '
'1666752572934,\n'
' "time" : "Wed Oct 26 04:49:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1087555072,\n'
' "io_kbytes" : 1062065,\n'
' "bw_bytes" : 155342818,\n'
' "bw" : 151701,\n'
' "iops" : 303402.942437,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2124124,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1522,\n'
' "max" : 29946,\n'
' "mean" : 2805.505978,\n'
' "stddev" : '
'1163.675313,\n'
' "N" : 2124124\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 190,\n'
' "max" : 125075,\n'
' "mean" : '
'23326.719890,\n'
' "stddev" : '
'2999.627077,\n'
' "N" : 2124131,\n'
' "percentile" : {\n'
' "1.000000" : 16768,\n'
' "5.000000" : 21376,\n'
' "10.000000" : '
'21632,\n'
' "20.000000" : '
'21888,\n'
' "30.000000" : '
'22656,\n'
' "40.000000" : '
'22912,\n'
' "50.000000" : '
'23168,\n'
' "60.000000" : '
'23168,\n'
' "70.000000" : '
'23680,\n'
' "80.000000" : '
'24192,\n'
' "90.000000" : '
'24448,\n'
' "95.000000" : '
'27264,\n'
' "99.000000" : '
'34560,\n'
' "99.500000" : '
'40192,\n'
' "99.900000" : '
'55040,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 77312\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6311,\n'
' "max" : 126648,\n'
' "mean" : '
'26132.225886,\n'
' "stddev" : '
'2927.082286,\n'
' "N" : 2124131\n'
' },\n'
' "bw_min" : 151465,\n'
' "bw_max" : 152209,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'151742.642857,\n'
' "bw_dev" : 193.711366,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 302930,\n'
' "iops_max" : 304420,\n'
' "iops_mean" : '
'303485.500000,\n'
' "iops_stddev" : '
'388.155550,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 31.928571,\n'
' "sys_cpu" : 68.042857,\n'
' "ctx" : 26,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999976,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.030177,\n'
' "20" : 2.877798,\n'
' "50" : 96.939209,\n'
' "100" : 0.136903,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=143673: Wed '
'Oct 26 04:49:32 2022\n'
' read: IOPS=303k, BW=148MiB/s '
'(155MB/s)(1037MiB/7001msec)\n'
' slat (nsec): min=1522, '
'max=29946, avg=2805.51, '
'stdev=1163.68\n'
' clat (nsec): min=190, '
'max=125075, avg=23326.72, '
'stdev=2999.63\n'
' lat (usec): min=6, max=126, '
'avg=26.13, stdev= 2.93\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16768], '
'5.00th=[21376], 10.00th=[21632], '
'20.00th=[21888],\n'
' | 30.00th=[22656], '
'40.00th=[22912], '
'50.00th=[23168], '
'60.00th=[23168],\n'
' | 70.00th=[23680], '
'80.00th=[24192], '
'90.00th=[24448], '
'95.00th=[27264],\n'
' | 99.00th=[34560], '
'99.50th=[40192], '
'99.90th=[55040], '
'99.95th=[65280],\n'
' | 99.99th=[77312]\n'
' bw ( KiB/s): min=151465, '
'max=152209, per=100.00%, '
'avg=151742.64, stdev=193.71, '
'samples=14\n'
' iops : min=302930, '
'max=304420, avg=303485.50, '
'stdev=388.16, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.03%, 20=2.88%, 50=96.94%, '
'100=0.14%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=31.93%, '
'sys=68.04%, ctx=26, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2124124,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=148MiB/s (155MB/s), '
'148MiB/s-148MiB/s '
'(155MB/s-155MB/s), io=1037MiB '
'(1088MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.output'),
'state': {'begin': 1666745483.8309026,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7939887046813965,
'end': 1666745484.6248913,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.state')},
'cmd_346': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.output'),
'state': {'begin': 1666745486.6498613,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7868752479553223,
'end': 1666745487.4367366,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.state')},
'cmd_347': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.output'),
'state': {'begin': 1666745487.4388833,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.456100225448608,
'end': 1666745498.8949835,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.state')},
'cmd_348': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752588,\n'
' "timestamp_ms" : '
'1666752588001,\n'
' "time" : "Wed Oct 26 04:49:48 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1087149056,\n'
' "io_kbytes" : 1061669,\n'
' "bw_bytes" : 155284824,\n'
' "bw" : 151645,\n'
' "iops" : 303289.530067,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2123330,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1523,\n'
' "max" : 35066,\n'
' "mean" : 2799.298077,\n'
' "stddev" : '
'1164.539197,\n'
' "N" : 2123331\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 200,\n'
' "max" : 131516,\n'
' "mean" : '
'23335.457212,\n'
' "stddev" : '
'2973.486735,\n'
' "N" : 2123337,\n'
' "percentile" : {\n'
' "1.000000" : 16768,\n'
' "5.000000" : 21376,\n'
' "10.000000" : '
'21632,\n'
' "20.000000" : '
'21888,\n'
' "30.000000" : '
'22656,\n'
' "40.000000" : '
'22912,\n'
' "50.000000" : '
'23168,\n'
' "60.000000" : '
'23168,\n'
' "70.000000" : '
'23680,\n'
' "80.000000" : '
'24192,\n'
' "90.000000" : '
'24448,\n'
' "95.000000" : '
'27008,\n'
' "99.000000" : '
'34560,\n'
' "99.500000" : '
'39680,\n'
' "99.900000" : '
'54528,\n'
' "99.950000" : '
'64768,\n'
' "99.990000" : 76288\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6322,\n'
' "max" : 135254,\n'
' "mean" : '
'26134.754886,\n'
' "stddev" : '
'2903.757258,\n'
' "N" : 2123337\n'
' },\n'
' "bw_min" : 151360,\n'
' "bw_max" : 152173,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'151693.142857,\n'
' "bw_dev" : 212.960761,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 302720,\n'
' "iops_max" : 304346,\n'
' "iops_mean" : '
'303386.500000,\n'
' "iops_stddev" : '
'425.766413,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 30.957143,\n'
' "sys_cpu" : 69.014286,\n'
' "ctx" : 24,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999976,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.029906,\n'
' "20" : 2.873694,\n'
' "50" : 96.944752,\n'
' "100" : 0.133187,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=143723: Wed '
'Oct 26 04:49:47 2022\n'
' read: IOPS=303k, BW=148MiB/s '
'(155MB/s)(1037MiB/7001msec)\n'
' slat (nsec): min=1523, '
'max=35066, avg=2799.30, '
'stdev=1164.54\n'
' clat (nsec): min=200, '
'max=131516, avg=23335.46, '
'stdev=2973.49\n'
' lat (usec): min=6, max=135, '
'avg=26.13, stdev= 2.90\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16768], '
'5.00th=[21376], 10.00th=[21632], '
'20.00th=[21888],\n'
' | 30.00th=[22656], '
'40.00th=[22912], '
'50.00th=[23168], '
'60.00th=[23168],\n'
' | 70.00th=[23680], '
'80.00th=[24192], '
'90.00th=[24448], '
'95.00th=[27008],\n'
' | 99.00th=[34560], '
'99.50th=[39680], '
'99.90th=[54528], '
'99.95th=[64768],\n'
' | 99.99th=[76288]\n'
' bw ( KiB/s): min=151360, '
'max=152173, per=100.00%, '
'avg=151693.14, stdev=212.96, '
'samples=14\n'
' iops : min=302720, '
'max=304346, avg=303386.50, '
'stdev=425.77, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.03%, 20=2.87%, 50=96.94%, '
'100=0.13%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=30.96%, '
'sys=69.01%, ctx=24, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2123330,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=148MiB/s (155MB/s), '
'148MiB/s-148MiB/s '
'(155MB/s-155MB/s), io=1037MiB '
'(1087MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.output'),
'state': {'begin': 1666745498.8982599,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7933619022369385,
'end': 1666745499.6916218,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.state')},
'cmd_349': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.output'),
'state': {'begin': 1666745501.7145822,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7818739414215088,
'end': 1666745502.4964561,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.state')},
'cmd_35': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.output'),
'state': {'begin': 1666743994.2055538,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="io_uring" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.037194967269897,
'end': 1666744005.2427487,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.state')},
'cmd_350': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.output'),
'state': {'begin': 1666745502.498419,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.457935333251953,
'end': 1666745513.9563544,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.state')},
'cmd_351': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752603,\n'
' "timestamp_ms" : '
'1666752603063,\n'
' "time" : "Wed Oct 26 04:50:03 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1081744896,\n'
' "io_kbytes" : 1056391,\n'
' "bw_bytes" : 154512911,\n'
' "bw" : 150891,\n'
' "iops" : 301782.031138,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2112776,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1522,\n'
' "max" : 27091,\n'
' "mean" : 2821.886822,\n'
' "stddev" : '
'1220.912718,\n'
' "N" : 2112776\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 190,\n'
' "max" : 120736,\n'
' "mean" : '
'23444.546176,\n'
' "stddev" : '
'2978.404735,\n'
' "N" : 2112783,\n'
' "percentile" : {\n'
' "1.000000" : 16768,\n'
' "5.000000" : 21376,\n'
' "10.000000" : '
'21632,\n'
' "20.000000" : '
'21888,\n'
' "30.000000" : '
'22656,\n'
' "40.000000" : '
'22912,\n'
' "50.000000" : '
'23168,\n'
' "60.000000" : '
'23424,\n'
' "70.000000" : '
'23680,\n'
' "80.000000" : '
'24192,\n'
' "90.000000" : '
'24704,\n'
' "95.000000" : '
'27008,\n'
' "99.000000" : '
'35072,\n'
' "99.500000" : '
'39680,\n'
' "99.900000" : '
'51968,\n'
' "99.950000" : '
'62208,\n'
' "99.990000" : 74240\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6622,\n'
' "max" : 122389,\n'
' "mean" : '
'26266.430108,\n'
' "stddev" : '
'2931.359656,\n'
' "N" : 2112783\n'
' },\n'
' "bw_min" : 150430,\n'
' "bw_max" : 151161,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'150901.538462,\n'
' "bw_dev" : 191.142013,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 300860,\n'
' "iops_max" : 302322,\n'
' "iops_mean" : '
'301803.076923,\n'
' "iops_stddev" : '
'382.467093,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 31.242857,\n'
' "sys_cpu" : 68.728571,\n'
' "ctx" : 25,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999976,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.029440,\n'
' "20" : 2.844267,\n'
' "50" : 96.992298,\n'
' "100" : 0.116103,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=143773: Wed '
'Oct 26 04:50:03 2022\n'
' read: IOPS=302k, BW=147MiB/s '
'(155MB/s)(1032MiB/7001msec)\n'
' slat (nsec): min=1522, '
'max=27091, avg=2821.89, '
'stdev=1220.91\n'
' clat (nsec): min=190, '
'max=120736, avg=23444.55, '
'stdev=2978.40\n'
' lat (usec): min=6, max=122, '
'avg=26.27, stdev= 2.93\n'
' clat percentiles (nsec):\n'
' | 1.00th=[16768], '
'5.00th=[21376], 10.00th=[21632], '
'20.00th=[21888],\n'
' | 30.00th=[22656], '
'40.00th=[22912], '
'50.00th=[23168], '
'60.00th=[23424],\n'
' | 70.00th=[23680], '
'80.00th=[24192], '
'90.00th=[24704], '
'95.00th=[27008],\n'
' | 99.00th=[35072], '
'99.50th=[39680], '
'99.90th=[51968], '
'99.95th=[62208],\n'
' | 99.99th=[74240]\n'
' bw ( KiB/s): min=150430, '
'max=151161, per=100.00%, '
'avg=150901.54, stdev=191.14, '
'samples=13\n'
' iops : min=300860, '
'max=302322, avg=301803.08, '
'stdev=382.47, samples=13\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=0.03%, 20=2.84%, 50=96.99%, '
'100=0.12%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=31.24%, '
'sys=68.73%, ctx=25, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2112776,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=147MiB/s (155MB/s), '
'147MiB/s-147MiB/s '
'(155MB/s-155MB/s), io=1032MiB '
'(1082MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.output'),
'state': {'begin': 1666745513.9593008,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7880733013153076,
'end': 1666745514.747374,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.state')},
'cmd_352': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.output'),
'state': {'begin': 1666745515.7417245,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7850258350372314,
'end': 1666745516.5267503,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.state')},
'cmd_353': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.output'),
'state': {'begin': 1666745516.5291083,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.059075117111206,
'end': 1666745527.5881834,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.state')},
'cmd_354': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752616,\n'
' "timestamp_ms" : '
'1666752616765,\n'
' "time" : "Wed Oct 26 04:50:16 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1090587136,\n'
' "io_kbytes" : 1065026,\n'
' "bw_bytes" : 155775908,\n'
' "bw" : 152124,\n'
' "iops" : 304248.678760,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2130045,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1382,\n'
' "max" : 75892,\n'
' "mean" : 2807.293279,\n'
' "stddev" : '
'1237.975793,\n'
' "N" : 2130046\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 110,\n'
' "max" : 140474,\n'
' "mean" : '
'23243.132478,\n'
' "stddev" : '
'4091.818656,\n'
' "N" : 2130052,\n'
' "percentile" : {\n'
' "1.000000" : 15424,\n'
' "5.000000" : 19328,\n'
' "10.000000" : '
'19584,\n'
' "20.000000" : '
'21120,\n'
' "30.000000" : '
'21632,\n'
' "40.000000" : '
'22656,\n'
' "50.000000" : '
'22912,\n'
' "60.000000" : '
'23168,\n'
' "70.000000" : '
'23936,\n'
' "80.000000" : '
'24704,\n'
' "90.000000" : '
'25984,\n'
' "95.000000" : '
'27264,\n'
' "99.000000" : '
'43264,\n'
' "99.500000" : '
'45824,\n'
' "99.900000" : '
'56576,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 81408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6923,\n'
' "max" : 141916,\n'
' "mean" : '
'26050.426103,\n'
' "stddev" : '
'4239.486761,\n'
' "N" : 2130052\n'
' },\n'
' "bw_min" : 151886,\n'
' "bw_max" : 152476,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'152167.357143,\n'
' "bw_dev" : 193.514304,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 303772,\n'
' "iops_max" : 304952,\n'
' "iops_mean" : '
'304334.714286,\n'
' "iops_stddev" : '
'386.864800,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 17.885714,\n'
' "sys_cpu" : 80.342857,\n'
' "ctx" : 11485,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999977,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.044882,\n'
' "20" : 15.144750,\n'
' "50" : 84.647883,\n'
' "100" : 0.147649,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=143812: Wed '
'Oct 26 04:50:16 2022\n'
' read: IOPS=304k, BW=149MiB/s '
'(156MB/s)(1040MiB/7001msec)\n'
' slat (nsec): min=1382, '
'max=75892, avg=2807.29, '
'stdev=1237.98\n'
' clat (nsec): min=110, '
'max=140474, avg=23243.13, '
'stdev=4091.82\n'
' lat (usec): min=6, max=141, '
'avg=26.05, stdev= 4.24\n'
' clat percentiles (nsec):\n'
' | 1.00th=[15424], '
'5.00th=[19328], 10.00th=[19584], '
'20.00th=[21120],\n'
' | 30.00th=[21632], '
'40.00th=[22656], '
'50.00th=[22912], '
'60.00th=[23168],\n'
' | 70.00th=[23936], '
'80.00th=[24704], '
'90.00th=[25984], '
'95.00th=[27264],\n'
' | 99.00th=[43264], '
'99.50th=[45824], '
'99.90th=[56576], '
'99.95th=[66048],\n'
' | 99.99th=[81408]\n'
' bw ( KiB/s): min=151886, '
'max=152476, per=100.00%, '
'avg=152167.36, stdev=193.51, '
'samples=14\n'
' iops : min=303772, '
'max=304952, avg=304334.71, '
'stdev=386.86, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.04%, 20=15.14%, '
'50=84.65%\n'
' lat (usec) : 100=0.15%, '
'250=0.01%\n'
' cpu : usr=17.89%, '
'sys=80.34%, ctx=11485, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2130045,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=149MiB/s (156MB/s), '
'149MiB/s-149MiB/s '
'(156MB/s-156MB/s), io=1040MiB '
'(1091MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.output'),
'state': {'begin': 1666745527.5914118,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7916409969329834,
'end': 1666745528.3830528,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.state')},
'cmd_355': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.output'),
'state': {'begin': 1666745529.3818274,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7902827262878418,
'end': 1666745530.17211,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.state')},
'cmd_356': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.output'),
'state': {'begin': 1666745530.1745954,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.050925016403198,
'end': 1666745541.2255204,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.state')},
'cmd_357': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752630,\n'
' "timestamp_ms" : '
'1666752630403,\n'
' "time" : "Wed Oct 26 04:50:30 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1091710976,\n'
' "io_kbytes" : 1066124,\n'
' "bw_bytes" : 155936434,\n'
' "bw" : 152281,\n'
' "iops" : 304562.205399,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2132240,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1372,\n'
' "max" : 65753,\n'
' "mean" : 2806.080467,\n'
' "stddev" : '
'1233.481209,\n'
' "N" : 2132241\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 110,\n'
' "max" : 142047,\n'
' "mean" : '
'23217.680802,\n'
' "stddev" : '
'4093.965853,\n'
' "N" : 2132247,\n'
' "percentile" : {\n'
' "1.000000" : 15424,\n'
' "5.000000" : 19328,\n'
' "10.000000" : '
'19584,\n'
' "20.000000" : '
'21120,\n'
' "30.000000" : '
'21632,\n'
' "40.000000" : '
'22656,\n'
' "50.000000" : '
'22912,\n'
' "60.000000" : '
'23168,\n'
' "70.000000" : '
'23936,\n'
' "80.000000" : '
'24704,\n'
' "90.000000" : '
'25984,\n'
' "95.000000" : '
'27264,\n'
' "99.000000" : '
'43264,\n'
' "99.500000" : '
'45824,\n'
' "99.900000" : '
'57600,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 81408\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6883,\n'
' "max" : 143439,\n'
' "mean" : '
'26023.761436,\n'
' "stddev" : '
'4231.145211,\n'
' "N" : 2132247\n'
' },\n'
' "bw_min" : 152213,\n'
' "bw_max" : 152540,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'152322.357143,\n'
' "bw_dev" : 94.171372,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 304426,\n'
' "iops_max" : 305080,\n'
' "iops_mean" : '
'304644.857143,\n'
' "iops_stddev" : '
'188.408254,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.142857,\n'
' "sys_cpu" : 79.042857,\n'
' "ctx" : 11717,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999977,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.046196,\n'
' "20" : 14.669643,\n'
' "50" : 85.119499,\n'
' "100" : 0.149655,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=143851: Wed '
'Oct 26 04:50:30 2022\n'
' read: IOPS=305k, BW=149MiB/s '
'(156MB/s)(1041MiB/7001msec)\n'
' slat (nsec): min=1372, '
'max=65753, avg=2806.08, '
'stdev=1233.48\n'
' clat (nsec): min=110, '
'max=142047, avg=23217.68, '
'stdev=4093.97\n'
' lat (usec): min=6, max=143, '
'avg=26.02, stdev= 4.23\n'
' clat percentiles (nsec):\n'
' | 1.00th=[15424], '
'5.00th=[19328], 10.00th=[19584], '
'20.00th=[21120],\n'
' | 30.00th=[21632], '
'40.00th=[22656], '
'50.00th=[22912], '
'60.00th=[23168],\n'
' | 70.00th=[23936], '
'80.00th=[24704], '
'90.00th=[25984], '
'95.00th=[27264],\n'
' | 99.00th=[43264], '
'99.50th=[45824], '
'99.90th=[57600], '
'99.95th=[67072],\n'
' | 99.99th=[81408]\n'
' bw ( KiB/s): min=152213, '
'max=152540, per=100.00%, '
'avg=152322.36, stdev=94.17, '
'samples=14\n'
' iops : min=304426, '
'max=305080, avg=304644.86, '
'stdev=188.41, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.05%, 20=14.67%, '
'50=85.12%\n'
' lat (usec) : 100=0.15%, '
'250=0.01%\n'
' cpu : usr=19.14%, '
'sys=79.04%, ctx=11717, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2132240,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=149MiB/s (156MB/s), '
'149MiB/s-149MiB/s '
'(156MB/s-156MB/s), io=1041MiB '
'(1092MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.output'),
'state': {'begin': 1666745541.2280989,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7909369468688965,
'end': 1666745542.0190358,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.state')},
'cmd_358': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.output'),
'state': {'begin': 1666745543.012986,
'cmd': 'rm '
'/tmp/fio-output_512_8_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7926218509674072,
'end': 1666745543.8056078,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.state')},
'cmd_359': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.output'),
'state': {'begin': 1666745543.807617,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_io_uring-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.05995512008667,
'end': 1666745554.867572,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.state')},
'cmd_36': {'output': 'io_uring-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=io_uring, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751094,\n'
' "timestamp_ms" : '
'1666751094444,\n'
' "time" : "Wed Oct 26 04:24:54 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "io_uring",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 286309376,\n'
' "io_kbytes" : 279599,\n'
' "bw_bytes" : 40895497,\n'
' "bw" : 39937,\n'
' "iops" : 79874.017997,\n'
' "runtime" : 7001,\n'
' "total_ios" : 559198,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2494,\n'
' "max" : 45485,\n'
' "mean" : 2642.445052,\n'
' "stddev" : 228.167226,\n'
' "N" : 559198\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 190,\n'
' "max" : 81473,\n'
' "mean" : 9223.417267,\n'
' "stddev" : '
'1276.987473,\n'
' "N" : 559198,\n'
' "percentile" : {\n'
' "1.000000" : 8768,\n'
' "5.000000" : 8896,\n'
' "10.000000" : 8896,\n'
' "20.000000" : 9024,\n'
' "30.000000" : 9024,\n'
' "40.000000" : 9024,\n'
' "50.000000" : 9024,\n'
' "60.000000" : 9024,\n'
' "70.000000" : 9152,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9152,\n'
' "95.000000" : 9408,\n'
' "99.000000" : 15424,\n'
' "99.500000" : 16192,\n'
' "99.900000" : 19584,\n'
' "99.950000" : 23168,\n'
' "99.990000" : 56576\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8957,\n'
' "max" : 84108,\n'
' "mean" : 11865.862319,\n'
' "stddev" : '
'1296.542780,\n'
' "N" : 559198\n'
' },\n'
' "bw_min" : 39762,\n'
' "bw_max" : 40171,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'39954.785714,\n'
' "bw_dev" : 110.705554,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 79524,\n'
' "iops_max" : 80342,\n'
' "iops_mean" : '
'79909.642857,\n'
' "iops_stddev" : '
'221.379454,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.328571,\n'
' "sys_cpu" : 46.328571,\n'
' "ctx" : 559054,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 97.150741,\n'
' "20" : 2.758057,\n'
' "50" : 0.062590,\n'
' "100" : 0.018062,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 801785,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5027,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5027,\n'
' "util" : 99.098564\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-reference: (groupid=0, '
'jobs=1): err= 0: pid=139136: Wed '
'Oct 26 04:24:54 2022\n'
' read: IOPS=79.9k, BW=39.0MiB/s '
'(40.9MB/s)(273MiB/7001msec)\n'
' slat (nsec): min=2494, '
'max=45485, avg=2642.45, '
'stdev=228.17\n'
' clat (nsec): min=190, '
'max=81473, avg=9223.42, '
'stdev=1276.99\n'
' lat (nsec): min=8957, '
'max=84108, avg=11865.86, '
'stdev=1296.54\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8768], 5.00th=[ '
'8896], 10.00th=[ 8896], 20.00th=[ '
'9024],\n'
' | 30.00th=[ 9024], 40.00th=[ '
'9024], 50.00th=[ 9024], 60.00th=[ '
'9024],\n'
' | 70.00th=[ 9152], 80.00th=[ '
'9152], 90.00th=[ 9152], 95.00th=[ '
'9408],\n'
' | 99.00th=[15424], '
'99.50th=[16192], 99.90th=[19584], '
'99.95th=[23168],\n'
' | 99.99th=[56576]\n'
' bw ( KiB/s): min=39762, '
'max=40171, per=100.00%, '
'avg=39954.79, stdev=110.71, '
'samples=14\n'
' iops : min=79524, '
'max=80342, avg=79909.64, '
'stdev=221.38, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 10=97.15%, '
'20=2.76%, 50=0.06%, 100=0.02%\n'
' cpu : usr=14.33%, '
'sys=46.33%, ctx=559054, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=559198,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=39.0MiB/s (40.9MB/s), '
'39.0MiB/s-39.0MiB/s '
'(40.9MB/s-40.9MB/s), io=273MiB '
'(286MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=801785/0, '
'merge=0/0, ticks=5027/0, '
'in_queue=5027, util=99.10%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.output'),
'state': {'begin': 1666744005.2492862,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7900209426879883,
'end': 1666744006.039307,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.state')},
'cmd_360': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752644,\n'
' "timestamp_ms" : '
'1666752644045,\n'
' "time" : "Wed Oct 26 04:50:44 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1097063424,\n'
' "io_kbytes" : 1071351,\n'
' "bw_bytes" : 156700960,\n'
' "bw" : 153028,\n'
' "iops" : 306055.420654,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2142694,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1362,\n'
' "max" : 43081,\n'
' "mean" : 2770.524205,\n'
' "stddev" : '
'1257.270481,\n'
' "N" : 2142695\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 110,\n'
' "max" : 142568,\n'
' "mean" : '
'23118.608042,\n'
' "stddev" : '
'3988.582641,\n'
' "N" : 2142701,\n'
' "percentile" : {\n'
' "1.000000" : 15424,\n'
' "5.000000" : 19328,\n'
' "10.000000" : '
'20864,\n'
' "20.000000" : '
'21120,\n'
' "30.000000" : '
'21376,\n'
' "40.000000" : '
'22400,\n'
' "50.000000" : '
'22656,\n'
' "60.000000" : '
'22912,\n'
' "70.000000" : '
'23680,\n'
' "80.000000" : '
'24192,\n'
' "90.000000" : '
'25472,\n'
' "95.000000" : '
'27776,\n'
' "99.000000" : '
'42752,\n'
' "99.500000" : '
'45824,\n'
' "99.900000" : '
'58624,\n'
' "99.950000" : '
'68096,\n'
' "99.990000" : 82432\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6923,\n'
' "max" : 143950,\n'
' "mean" : '
'25889.131866,\n'
' "stddev" : '
'4007.357932,\n'
' "N" : 2142701\n'
' },\n'
' "bw_min" : 152809,\n'
' "bw_max" : 153283,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'153068.142857,\n'
' "bw_dev" : 168.665740,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 305618,\n'
' "iops_max" : 306566,\n'
' "iops_mean" : '
'306136.214286,\n'
' "iops_stddev" : '
'337.609738,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 19.928571,\n'
' "sys_cpu" : 78.200000,\n'
' "ctx" : 12004,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999977,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 0.040043,\n'
' "20" : 8.114318,\n'
' "50" : 91.677533,\n'
' "100" : 0.156905,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=143892: Wed '
'Oct 26 04:50:44 2022\n'
' read: IOPS=306k, BW=149MiB/s '
'(157MB/s)(1046MiB/7001msec)\n'
' slat (nsec): min=1362, '
'max=43081, avg=2770.52, '
'stdev=1257.27\n'
' clat (nsec): min=110, '
'max=142568, avg=23118.61, '
'stdev=3988.58\n'
' lat (usec): min=6, max=143, '
'avg=25.89, stdev= 4.01\n'
' clat percentiles (nsec):\n'
' | 1.00th=[15424], '
'5.00th=[19328], 10.00th=[20864], '
'20.00th=[21120],\n'
' | 30.00th=[21376], '
'40.00th=[22400], '
'50.00th=[22656], '
'60.00th=[22912],\n'
' | 70.00th=[23680], '
'80.00th=[24192], '
'90.00th=[25472], '
'95.00th=[27776],\n'
' | 99.00th=[42752], '
'99.50th=[45824], '
'99.90th=[58624], '
'99.95th=[68096],\n'
' | 99.99th=[82432]\n'
' bw ( KiB/s): min=152809, '
'max=153283, per=100.00%, '
'avg=153068.14, stdev=168.67, '
'samples=14\n'
' iops : min=305618, '
'max=306566, avg=306136.21, '
'stdev=337.61, samples=14\n'
' lat (nsec) : 250=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=0.04%, 20=8.11%, '
'50=91.68%\n'
' lat (usec) : 100=0.16%, '
'250=0.01%\n'
' cpu : usr=19.93%, '
'sys=78.20%, ctx=12004, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2142694,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=149MiB/s (157MB/s), '
'149MiB/s-149MiB/s '
'(157MB/s-157MB/s), io=1046MiB '
'(1097MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.output'),
'state': {'begin': 1666745554.8706086,
'cmd': 'cat '
'/tmp/fio-output_512_8_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7866778373718262,
'end': 1666745555.6572864,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.state')},
'cmd_361': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.output'),
'state': {'begin': 1666745556.657499,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.8002305030822754,
'end': 1666745557.4577296,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.state')},
'cmd_362': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.output'),
'state': {'begin': 1666745557.45975,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.076076030731201,
'end': 1666745568.535826,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.state')},
'cmd_363': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752657,\n'
' "timestamp_ms" : '
'1666752657713,\n'
' "time" : "Wed Oct 26 04:50:57 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1046272512,\n'
' "io_kbytes" : 1021750,\n'
' "bw_bytes" : 149446152,\n'
' "bw" : 145943,\n'
' "iops" : 291885.873447,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2043493,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1402,\n'
' "max" : 113964,\n'
' "mean" : 2707.785157,\n'
' "stddev" : '
'1337.154939,\n'
' "N" : 2043494\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2595,\n'
' "max" : 138590,\n'
' "mean" : '
'24451.180669,\n'
' "stddev" : '
'3177.840339,\n'
' "N" : 2043500,\n'
' "percentile" : {\n'
' "1.000000" : 17536,\n'
' "5.000000" : 22400,\n'
' "10.000000" : '
'22656,\n'
' "20.000000" : '
'22656,\n'
' "30.000000" : '
'22912,\n'
' "40.000000" : '
'23680,\n'
' "50.000000" : '
'24192,\n'
' "60.000000" : '
'24448,\n'
' "70.000000" : '
'25216,\n'
' "80.000000" : '
'25472,\n'
' "90.000000" : '
'25728,\n'
' "95.000000" : '
'29312,\n'
' "99.000000" : '
'37632,\n'
' "99.500000" : '
'42240,\n'
' "99.900000" : '
'54528,\n'
' "99.950000" : '
'67072,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10450,\n'
' "max" : 143930,\n'
' "mean" : '
'27158.965725,\n'
' "stddev" : '
'3022.018116,\n'
' "N" : 2043500\n'
' },\n'
' "bw_min" : 145819,\n'
' "bw_max" : 146239,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'146001.692308,\n'
' "bw_dev" : 141.640028,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 291638,\n'
' "iops_max" : 292478,\n'
' "iops_mean" : '
'292003.461538,\n'
' "iops_stddev" : '
'283.176216,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 35.971429,\n'
' "sys_cpu" : 62.771429,\n'
' "ctx" : 10182,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999951,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 2.033870,\n'
' "50" : 97.828033,\n'
' "100" : 0.127282,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2858731,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 15931,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 15931,\n'
' "util" : 99.035239\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=143932: Wed '
'Oct 26 04:50:57 2022\n'
' read: IOPS=292k, BW=143MiB/s '
'(149MB/s)(998MiB/7001msec)\n'
' slat (nsec): min=1402, '
'max=113964, avg=2707.79, '
'stdev=1337.15\n'
' clat (usec): min=2, max=138, '
'avg=24.45, stdev= 3.18\n'
' lat (usec): min=10, '
'max=143, avg=27.16, stdev= 3.02\n'
' clat percentiles (nsec):\n'
' | 1.00th=[17536], '
'5.00th=[22400], 10.00th=[22656], '
'20.00th=[22656],\n'
' | 30.00th=[22912], '
'40.00th=[23680], '
'50.00th=[24192], '
'60.00th=[24448],\n'
' | 70.00th=[25216], '
'80.00th=[25472], '
'90.00th=[25728], '
'95.00th=[29312],\n'
' | 99.00th=[37632], '
'99.50th=[42240], '
'99.90th=[54528], '
'99.95th=[67072],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=145819, '
'max=146239, per=100.00%, '
'avg=146001.69, stdev=141.64, '
'samples=13\n'
' iops : min=291638, '
'max=292478, avg=292003.46, '
'stdev=283.18, samples=13\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=2.03%, 50=97.83%, '
'100=0.13%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=35.97%, '
'sys=62.77%, ctx=10182, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2043493,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=143MiB/s (149MB/s), '
'143MiB/s-143MiB/s '
'(149MB/s-149MB/s), io=998MiB '
'(1046MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2858731/0, '
'merge=0/0, ticks=15931/0, '
'in_queue=15931, util=99.04%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.output'),
'state': {'begin': 1666745568.538666,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7850360870361328,
'end': 1666745569.323702,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.state')},
'cmd_364': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.output'),
'state': {'begin': 1666745570.3126845,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7866883277893066,
'end': 1666745571.0993729,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.state')},
'cmd_365': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.output'),
'state': {'begin': 1666745571.1018023,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.070146083831787,
'end': 1666745582.1719484,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.state')},
'cmd_366': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752671,\n'
' "timestamp_ms" : '
'1666752671349,\n'
' "time" : "Wed Oct 26 04:51:11 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1046752768,\n'
' "io_kbytes" : 1022219,\n'
' "bw_bytes" : 149514750,\n'
' "bw" : 146010,\n'
' "iops" : 292019.854307,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2044431,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1403,\n'
' "max" : 123251,\n'
' "mean" : 2715.633511,\n'
' "stddev" : '
'1330.313775,\n'
' "N" : 2044432\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 2605,\n'
' "max" : 145643,\n'
' "mean" : '
'24432.144750,\n'
' "stddev" : '
'3169.438036,\n'
' "N" : 2044438,\n'
' "percentile" : {\n'
' "1.000000" : 17536,\n'
' "5.000000" : 22400,\n'
' "10.000000" : '
'22656,\n'
' "20.000000" : '
'22656,\n'
' "30.000000" : '
'22912,\n'
' "40.000000" : '
'23680,\n'
' "50.000000" : '
'23936,\n'
' "60.000000" : '
'24448,\n'
' "70.000000" : '
'24960,\n'
' "80.000000" : '
'25472,\n'
' "90.000000" : '
'25728,\n'
' "95.000000" : '
'29312,\n'
' "99.000000" : '
'37632,\n'
' "99.500000" : '
'41728,\n'
' "99.900000" : '
'52992,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10781,\n'
' "max" : 148709,\n'
' "mean" : '
'27147.777613,\n'
' "stddev" : '
'3031.587372,\n'
' "N" : 2044438\n'
' },\n'
' "bw_min" : 145798,\n'
' "bw_max" : 146766,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'146077.000000,\n'
' "bw_dev" : 239.693394,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 291596,\n'
' "iops_max" : 293532,\n'
' "iops_mean" : '
'292154.142857,\n'
' "iops_stddev" : '
'479.286466,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 38.614286,\n'
' "sys_cpu" : 60.157143,\n'
' "ctx" : 10093,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999951,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 0.010000,\n'
' "20" : 2.060720,\n'
' "50" : 97.807507,\n'
' "100" : 0.119887,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2933310,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 16349,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 16349,\n'
' "util" : 99.098564\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=143970: Wed '
'Oct 26 04:51:11 2022\n'
' read: IOPS=292k, BW=143MiB/s '
'(150MB/s)(998MiB/7001msec)\n'
' slat (nsec): min=1403, '
'max=123251, avg=2715.63, '
'stdev=1330.31\n'
' clat (usec): min=2, max=145, '
'avg=24.43, stdev= 3.17\n'
' lat (usec): min=10, '
'max=148, avg=27.15, stdev= 3.03\n'
' clat percentiles (nsec):\n'
' | 1.00th=[17536], '
'5.00th=[22400], 10.00th=[22656], '
'20.00th=[22656],\n'
' | 30.00th=[22912], '
'40.00th=[23680], '
'50.00th=[23936], '
'60.00th=[24448],\n'
' | 70.00th=[24960], '
'80.00th=[25472], '
'90.00th=[25728], '
'95.00th=[29312],\n'
' | 99.00th=[37632], '
'99.50th=[41728], '
'99.90th=[52992], '
'99.95th=[65280],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=145798, '
'max=146766, per=100.00%, '
'avg=146077.00, stdev=239.69, '
'samples=14\n'
' iops : min=291596, '
'max=293532, avg=292154.14, '
'stdev=479.29, samples=14\n'
' lat (usec) : 4=0.01%, '
'10=0.01%, 20=2.06%, 50=97.81%, '
'100=0.12%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=38.61%, '
'sys=60.16%, ctx=10093, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2044431,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=143MiB/s (150MB/s), '
'143MiB/s-143MiB/s '
'(150MB/s-150MB/s), io=998MiB '
'(1047MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2933310/0, '
'merge=0/0, ticks=16349/0, '
'in_queue=16349, util=99.10%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.output'),
'state': {'begin': 1666745582.1785822,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.791018009185791,
'end': 1666745582.9696002,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.state')},
'cmd_367': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.output'),
'state': {'begin': 1666745583.9647493,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7866001129150391,
'end': 1666745584.7513494,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.state')},
'cmd_368': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.output'),
'state': {'begin': 1666745584.753581,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.070449590682983,
'end': 1666745595.8240306,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.state')},
'cmd_369': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752685,\n'
' "timestamp_ms" : '
'1666752685000,\n'
' "time" : "Wed Oct 26 04:51:25 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1043872256,\n'
' "io_kbytes" : 1019406,\n'
' "bw_bytes" : 149103307,\n'
' "bw" : 145608,\n'
' "iops" : 291216.254821,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2038805,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1432,\n'
' "max" : 27141,\n'
' "mean" : 2732.441553,\n'
' "stddev" : '
'1335.007083,\n'
' "N" : 2038806\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4679,\n'
' "max" : 159368,\n'
' "mean" : '
'24492.867077,\n'
' "stddev" : '
'3138.428213,\n'
' "N" : 2038812,\n'
' "percentile" : {\n'
' "1.000000" : 17536,\n'
' "5.000000" : 22400,\n'
' "10.000000" : '
'22656,\n'
' "20.000000" : '
'22912,\n'
' "30.000000" : '
'22912,\n'
' "40.000000" : '
'23680,\n'
' "50.000000" : '
'24192,\n'
' "60.000000" : '
'24960,\n'
' "70.000000" : '
'25216,\n'
' "80.000000" : '
'25472,\n'
' "90.000000" : '
'25728,\n'
' "95.000000" : '
'29312,\n'
' "99.000000" : '
'37632,\n'
' "99.500000" : '
'41728,\n'
' "99.900000" : '
'52480,\n'
' "99.950000" : '
'65280,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10911,\n'
' "max" : 160841,\n'
' "mean" : '
'27225.311592,\n'
' "stddev" : '
'2968.109528,\n'
' "N" : 2038812\n'
' },\n'
' "bw_min" : 145239,\n'
' "bw_max" : 145957,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'145648.230769,\n'
' "bw_dev" : 185.987972,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 290478,\n'
' "iops_max" : 291914,\n'
' "iops_mean" : '
'291296.538462,\n'
' "iops_stddev" : '
'372.038218,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 37.700000,\n'
' "sys_cpu" : 61.085714,\n'
' "ctx" : 10154,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999951,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 1.971253,\n'
' "50" : 97.900584,\n'
' "100" : 0.117912,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 2855695,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 15886,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 15886,\n'
' "util" : 99.045298\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=144008: Wed '
'Oct 26 04:51:24 2022\n'
' read: IOPS=291k, BW=142MiB/s '
'(149MB/s)(996MiB/7001msec)\n'
' slat (nsec): min=1432, '
'max=27141, avg=2732.44, '
'stdev=1335.01\n'
' clat (usec): min=4, max=159, '
'avg=24.49, stdev= 3.14\n'
' lat (usec): min=10, '
'max=160, avg=27.23, stdev= 2.97\n'
' clat percentiles (nsec):\n'
' | 1.00th=[17536], '
'5.00th=[22400], 10.00th=[22656], '
'20.00th=[22912],\n'
' | 30.00th=[22912], '
'40.00th=[23680], '
'50.00th=[24192], '
'60.00th=[24960],\n'
' | 70.00th=[25216], '
'80.00th=[25472], '
'90.00th=[25728], '
'95.00th=[29312],\n'
' | 99.00th=[37632], '
'99.50th=[41728], '
'99.90th=[52480], '
'99.95th=[65280],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=145239, '
'max=145957, per=100.00%, '
'avg=145648.23, stdev=185.99, '
'samples=13\n'
' iops : min=290478, '
'max=291914, avg=291296.54, '
'stdev=372.04, samples=13\n'
' lat (usec) : 10=0.01%, '
'20=1.97%, 50=97.90%, 100=0.12%, '
'250=0.01%\n'
' cpu : usr=37.70%, '
'sys=61.09%, ctx=10154, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2038805,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=142MiB/s (149MB/s), '
'142MiB/s-142MiB/s '
'(149MB/s-149MB/s), io=996MiB '
'(1044MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=2855695/0, '
'merge=0/0, ticks=15886/0, '
'in_queue=15886, util=99.05%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.output'),
'state': {'begin': 1666745595.8270845,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7876555919647217,
'end': 1666745596.6147401,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.state')},
'cmd_37': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.output'),
'state': {'begin': 1666744008.0577302,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.7847611904144287,
'end': 1666744008.8424914,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.state')},
'cmd_370': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.output'),
'state': {'begin': 1666745598.6471202,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7935197353363037,
'end': 1666745599.44064,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.state')},
'cmd_371': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.output'),
'state': {'begin': 1666745599.4430456,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_aio_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.500689506530762,
'end': 1666745610.9437351,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.state')},
'cmd_372': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752700,\n'
' "timestamp_ms" : '
'1666752700018,\n'
' "time" : "Wed Oct 26 04:51:40 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1086418944,\n'
' "io_kbytes" : 1060956,\n'
' "bw_bytes" : 155180537,\n'
' "bw" : 151543,\n'
' "iops" : 303086.559063,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2121909,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1593,\n'
' "max" : 35116,\n'
' "mean" : 2769.186245,\n'
' "stddev" : '
'1002.279645,\n'
' "N" : 2121910\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 330,\n'
' "max" : 188353,\n'
' "mean" : '
'14078.489138,\n'
' "stddev" : '
'6683.083761,\n'
' "N" : 2121912,\n'
' "percentile" : {\n'
' "1.000000" : 3408,\n'
' "5.000000" : 3536,\n'
' "10.000000" : 4832,\n'
' "20.000000" : 6816,\n'
' "30.000000" : 9664,\n'
' "40.000000" : '
'12096,\n'
' "50.000000" : '
'14272,\n'
' "60.000000" : '
'16320,\n'
' "70.000000" : '
'18816,\n'
' "80.000000" : '
'20608,\n'
' "90.000000" : '
'22912,\n'
' "95.000000" : '
'23424,\n'
' "99.000000" : '
'27264,\n'
' "99.500000" : '
'29824,\n'
' "99.900000" : '
'42752,\n'
' "99.950000" : '
'57088,\n'
' "99.990000" : 70144\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6182,\n'
' "max" : 191369,\n'
' "mean" : '
'16847.674926,\n'
' "stddev" : '
'6639.843683,\n'
' "N" : 2121912\n'
' },\n'
' "bw_min" : 150945,\n'
' "bw_max" : 152439,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'151590.285714,\n'
' "bw_dev" : 485.280320,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 301890,\n'
' "iops_max" : 304878,\n'
' "iops_mean" : '
'303180.571429,\n'
' "iops_stddev" : '
'970.850675,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 40.557143,\n'
' "sys_cpu" : 59.414286,\n'
' "ctx" : 26,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.471274,\n'
' "2" : 26.285576,\n'
' "4" : 58.295997,\n'
' "8" : 14.947154,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 87.443970,\n'
' "8" : 12.556030,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.033743,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 6.563524,\n'
' "10" : 29.667955,\n'
' "20" : 42.265856,\n'
' "50" : 21.394131,\n'
' "100" : 0.072718,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=144056: Wed '
'Oct 26 04:51:40 2022\n'
' read: IOPS=303k, BW=148MiB/s '
'(155MB/s)(1036MiB/7001msec)\n'
' slat (nsec): min=1593, '
'max=35116, avg=2769.19, '
'stdev=1002.28\n'
' clat (nsec): min=330, '
'max=188353, avg=14078.49, '
'stdev=6683.08\n'
' lat (usec): min=6, max=191, '
'avg=16.85, stdev= 6.64\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3408], '
'5.00th=[ 3536], 10.00th=[ 4832], '
'20.00th=[ 6816],\n'
' | 30.00th=[ 9664], '
'40.00th=[12096], '
'50.00th=[14272], '
'60.00th=[16320],\n'
' | 70.00th=[18816], '
'80.00th=[20608], '
'90.00th=[22912], '
'95.00th=[23424],\n'
' | 99.00th=[27264], '
'99.50th=[29824], '
'99.90th=[42752], '
'99.95th=[57088],\n'
' | 99.99th=[70144]\n'
' bw ( KiB/s): min=150945, '
'max=152439, per=100.00%, '
'avg=151590.29, stdev=485.28, '
'samples=14\n'
' iops : min=301890, '
'max=304878, avg=303180.57, '
'stdev=970.85, samples=14\n'
' lat (nsec) : 500=0.03%, '
'750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=6.56%, 10=29.67%, 20=42.27%, '
'50=21.39%\n'
' lat (usec) : 100=0.07%, '
'250=0.01%\n'
' cpu : usr=40.56%, '
'sys=59.41%, ctx=26, majf=0, '
'minf=0\n'
' IO depths : 1=0.5%, '
'2=26.3%, 4=58.3%, 8=14.9%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=87.4%, 8=12.6%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2121909,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=148MiB/s (155MB/s), '
'148MiB/s-148MiB/s '
'(155MB/s-155MB/s), io=1036MiB '
'(1086MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.output'),
'state': {'begin': 1666745610.9466832,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7992534637451172,
'end': 1666745611.7459366,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.state')},
'cmd_373': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.output'),
'state': {'begin': 1666745613.8330824,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7890000343322754,
'end': 1666745614.6220825,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.state')},
'cmd_374': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.output'),
'state': {'begin': 1666745614.6243002,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_aio_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.417046070098877,
'end': 1666745626.0413463,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.state')},
'cmd_375': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752715,\n'
' "timestamp_ms" : '
'1666752715116,\n'
' "time" : "Wed Oct 26 04:51:55 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1076485120,\n'
' "io_kbytes" : 1051255,\n'
' "bw_bytes" : 153761622,\n'
' "bw" : 150157,\n'
' "iops" : 300314.669333,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2102503,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1593,\n'
' "max" : 36158,\n'
' "mean" : 2832.892318,\n'
' "stddev" : '
'1021.358270,\n'
' "N" : 2102503\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 350,\n'
' "max" : 128501,\n'
' "mean" : '
'14078.870878,\n'
' "stddev" : '
'6722.192899,\n'
' "N" : 2102503,\n'
' "percentile" : {\n'
' "1.000000" : 3440,\n'
' "5.000000" : 3568,\n'
' "10.000000" : 4832,\n'
' "20.000000" : 6816,\n'
' "30.000000" : 9664,\n'
' "40.000000" : '
'11712,\n'
' "50.000000" : '
'14144,\n'
' "60.000000" : '
'16320,\n'
' "70.000000" : '
'18304,\n'
' "80.000000" : '
'19840,\n'
' "90.000000" : '
'23168,\n'
' "95.000000" : '
'23680,\n'
' "99.000000" : '
'27008,\n'
' "99.500000" : '
'29824,\n'
' "99.900000" : '
'44800,\n'
' "99.950000" : '
'57600,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6161,\n'
' "max" : 130174,\n'
' "mean" : '
'16911.763196,\n'
' "stddev" : '
'6736.191451,\n'
' "N" : 2102503\n'
' },\n'
' "bw_min" : 149226,\n'
' "bw_max" : 150989,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'150194.785714,\n'
' "bw_dev" : 508.754464,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 298452,\n'
' "iops_max" : 301979,\n'
' "iops_mean" : '
'300389.642857,\n'
' "iops_stddev" : '
'1017.629040,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 40.442857,\n'
' "sys_cpu" : 59.528571,\n'
' "ctx" : 29,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.466064,\n'
' "2" : 26.752542,\n'
' "4" : 57.926671,\n'
' "8" : 14.854723,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 87.516984,\n'
' "8" : 12.483016,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.042806,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 6.502393,\n'
' "10" : 29.740600,\n'
' "20" : 43.938059,\n'
' "50" : 19.695097,\n'
' "100" : 0.078906,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=144105: Wed '
'Oct 26 04:51:55 2022\n'
' read: IOPS=300k, BW=147MiB/s '
'(154MB/s)(1027MiB/7001msec)\n'
' slat (nsec): min=1593, '
'max=36158, avg=2832.89, '
'stdev=1021.36\n'
' clat (nsec): min=350, '
'max=128501, avg=14078.87, '
'stdev=6722.19\n'
' lat (usec): min=6, max=130, '
'avg=16.91, stdev= 6.74\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3440], '
'5.00th=[ 3568], 10.00th=[ 4832], '
'20.00th=[ 6816],\n'
' | 30.00th=[ 9664], '
'40.00th=[11712], '
'50.00th=[14144], '
'60.00th=[16320],\n'
' | 70.00th=[18304], '
'80.00th=[19840], '
'90.00th=[23168], '
'95.00th=[23680],\n'
' | 99.00th=[27008], '
'99.50th=[29824], '
'99.90th=[44800], '
'99.95th=[57600],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=149226, '
'max=150989, per=100.00%, '
'avg=150194.79, stdev=508.75, '
'samples=14\n'
' iops : min=298452, '
'max=301979, avg=300389.64, '
'stdev=1017.63, samples=14\n'
' lat (nsec) : 500=0.04%, '
'750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=6.50%, 10=29.74%, 20=43.94%, '
'50=19.70%\n'
' lat (usec) : 100=0.08%, '
'250=0.01%\n'
' cpu : usr=40.44%, '
'sys=59.53%, ctx=29, majf=0, '
'minf=0\n'
' IO depths : 1=0.5%, '
'2=26.8%, 4=57.9%, 8=14.9%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=87.5%, 8=12.5%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2102503,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=147MiB/s (154MB/s), '
'147MiB/s-147MiB/s '
'(154MB/s-154MB/s), io=1027MiB '
'(1076MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.output'),
'state': {'begin': 1666745626.044057,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7855679988861084,
'end': 1666745626.829625,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.state')},
'cmd_376': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.output'),
'state': {'begin': 1666745628.8473525,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7852327823638916,
'end': 1666745629.6325853,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.state')},
'cmd_377': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.output'),
'state': {'begin': 1666745629.6349015,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_aio_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.413702487945557,
'end': 1666745641.048604,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.state')},
'cmd_378': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752730,\n'
' "timestamp_ms" : '
'1666752730123,\n'
' "time" : "Wed Oct 26 04:52:10 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1091307008,\n'
' "io_kbytes" : 1065729,\n'
' "bw_bytes" : 155878732,\n'
' "bw" : 152225,\n'
' "iops" : 304450.078560,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2131455,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1572,\n'
' "max" : 80341,\n'
' "mean" : 2723.375660,\n'
' "stddev" : '
'926.510920,\n'
' "N" : 2131456\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 320,\n'
' "max" : 145703,\n'
' "mean" : '
'14031.642536,\n'
' "stddev" : '
'6669.051166,\n'
' "N" : 2131459,\n'
' "percentile" : {\n'
' "1.000000" : 3376,\n'
' "5.000000" : 3504,\n'
' "10.000000" : 4832,\n'
' "20.000000" : 6688,\n'
' "30.000000" : 9536,\n'
' "40.000000" : '
'11968,\n'
' "50.000000" : '
'14144,\n'
' "60.000000" : '
'16512,\n'
' "70.000000" : '
'18560,\n'
' "80.000000" : '
'20352,\n'
' "90.000000" : '
'22912,\n'
' "95.000000" : '
'23424,\n'
' "99.000000" : '
'27008,\n'
' "99.500000" : '
'29312,\n'
' "99.900000" : '
'45312,\n'
' "99.950000" : '
'58624,\n'
' "99.990000" : 70144\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6122,\n'
' "max" : 147336,\n'
' "mean" : '
'16755.017869,\n'
' "stddev" : '
'6618.416459,\n'
' "N" : 2131459\n'
' },\n'
' "bw_min" : 151543,\n'
' "bw_max" : 152997,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'152264.642857,\n'
' "bw_dev" : 377.366300,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 303086,\n'
' "iops_max" : 305994,\n'
' "iops_mean" : '
'304529.357143,\n'
' "iops_stddev" : '
'754.613463,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 41.871429,\n'
' "sys_cpu" : 58.100000,\n'
' "ctx" : 26,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.501066,\n'
' "2" : 26.540462,\n'
' "4" : 58.078777,\n'
' "8" : 14.879695,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 87.483879,\n'
' "8" : 12.516121,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.044148,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 6.314982,\n'
' "10" : 29.727393,\n'
' "20" : 43.166475,\n'
' "50" : 20.663397,\n'
' "100" : 0.081963,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=144155: Wed '
'Oct 26 04:52:10 2022\n'
' read: IOPS=304k, BW=149MiB/s '
'(156MB/s)(1041MiB/7001msec)\n'
' slat (nsec): min=1572, '
'max=80341, avg=2723.38, '
'stdev=926.51\n'
' clat (nsec): min=320, '
'max=145703, avg=14031.64, '
'stdev=6669.05\n'
' lat (usec): min=6, max=147, '
'avg=16.76, stdev= 6.62\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3376], '
'5.00th=[ 3504], 10.00th=[ 4832], '
'20.00th=[ 6688],\n'
' | 30.00th=[ 9536], '
'40.00th=[11968], '
'50.00th=[14144], '
'60.00th=[16512],\n'
' | 70.00th=[18560], '
'80.00th=[20352], '
'90.00th=[22912], '
'95.00th=[23424],\n'
' | 99.00th=[27008], '
'99.50th=[29312], '
'99.90th=[45312], '
'99.95th=[58624],\n'
' | 99.99th=[70144]\n'
' bw ( KiB/s): min=151543, '
'max=152997, per=100.00%, '
'avg=152264.64, stdev=377.37, '
'samples=14\n'
' iops : min=303086, '
'max=305994, avg=304529.36, '
'stdev=754.61, samples=14\n'
' lat (nsec) : 500=0.04%, '
'750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=6.31%, 10=29.73%, 20=43.17%, '
'50=20.66%\n'
' lat (usec) : 100=0.08%, '
'250=0.01%\n'
' cpu : usr=41.87%, '
'sys=58.10%, ctx=26, majf=0, '
'minf=0\n'
' IO depths : 1=0.5%, '
'2=26.5%, 4=58.1%, 8=14.9%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=87.5%, 8=12.5%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2131455,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=149MiB/s (156MB/s), '
'149MiB/s-149MiB/s '
'(156MB/s-156MB/s), io=1041MiB '
'(1091MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.output'),
'state': {'begin': 1666745641.0516608,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.8644020557403564,
'end': 1666745641.9160628,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.state')},
'cmd_379': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.output'),
'state': {'begin': 1666745644.222322,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8765056133270264,
'end': 1666745645.0988276,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.state')},
'cmd_38': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.output'),
'state': {'begin': 1666744008.8447871,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.455432415008545,
'end': 1666744020.3002195,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.state')},
'cmd_380': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.output'),
'state': {'begin': 1666745645.1010451,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.647058725357056,
'end': 1666745656.7481039,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.state')},
'cmd_381': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752745,\n'
' "timestamp_ms" : '
'1666752745825,\n'
' "time" : "Wed Oct 26 04:52:25 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1052019200,\n'
' "io_kbytes" : 1027362,\n'
' "bw_bytes" : 150266990,\n'
' "bw" : 146745,\n'
' "iops" : 293489.072990,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2054717,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1623,\n'
' "max" : 44804,\n'
' "mean" : 2930.171240,\n'
' "stddev" : '
'1093.681472,\n'
' "N" : 2054718\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 751,\n'
' "max" : 124714,\n'
' "mean" : '
'14597.804246,\n'
' "stddev" : '
'6757.196878,\n'
' "N" : 2054717,\n'
' "percentile" : {\n'
' "1.000000" : 3920,\n'
' "5.000000" : 4048,\n'
' "10.000000" : 5280,\n'
' "20.000000" : 8256,\n'
' "30.000000" : '
'10176,\n'
' "40.000000" : '
'12096,\n'
' "50.000000" : '
'14656,\n'
' "60.000000" : '
'16768,\n'
' "70.000000" : '
'18560,\n'
' "80.000000" : '
'20096,\n'
' "90.000000" : '
'24192,\n'
' "95.000000" : '
'24448,\n'
' "99.000000" : '
'27264,\n'
' "99.500000" : '
'30336,\n'
' "99.900000" : '
'42752,\n'
' "99.950000" : '
'56576,\n'
' "99.990000" : 69120\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6472,\n'
' "max" : 126397,\n'
' "mean" : '
'17527.975420,\n'
' "stddev" : '
'6799.693481,\n'
' "N" : 2054717\n'
' },\n'
' "bw_min" : 146440,\n'
' "bw_max" : 147365,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'146789.214286,\n'
' "bw_dev" : 284.407935,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 292880,\n'
' "iops_max" : 294730,\n'
' "iops_mean" : '
'293578.500000,\n'
' "iops_stddev" : '
'568.809925,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.014286,\n'
' "sys_cpu" : 60.957143,\n'
' "ctx" : 27,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.407453,\n'
' "2" : 26.810310,\n'
' "4" : 57.959417,\n'
' "8" : 14.822820,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 87.512790,\n'
' "8" : 12.487210,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.029931\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 3.778185,\n'
' "10" : 23.005261,\n'
' "20" : 50.434439,\n'
' "50" : 22.678257,\n'
' "100" : 0.072613,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=144204: Wed '
'Oct 26 04:52:25 2022\n'
' read: IOPS=293k, BW=143MiB/s '
'(150MB/s)(1003MiB/7001msec)\n'
' slat (nsec): min=1623, '
'max=44804, avg=2930.17, '
'stdev=1093.68\n'
' clat (nsec): min=751, '
'max=124714, avg=14597.80, '
'stdev=6757.20\n'
' lat (usec): min=6, max=126, '
'avg=17.53, stdev= 6.80\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3920], '
'5.00th=[ 4048], 10.00th=[ 5280], '
'20.00th=[ 8256],\n'
' | 30.00th=[10176], '
'40.00th=[12096], '
'50.00th=[14656], '
'60.00th=[16768],\n'
' | 70.00th=[18560], '
'80.00th=[20096], '
'90.00th=[24192], '
'95.00th=[24448],\n'
' | 99.00th=[27264], '
'99.50th=[30336], '
'99.90th=[42752], '
'99.95th=[56576],\n'
' | 99.99th=[69120]\n'
' bw ( KiB/s): min=146440, '
'max=147365, per=100.00%, '
'avg=146789.21, stdev=284.41, '
'samples=14\n'
' iops : min=292880, '
'max=294730, avg=293578.50, '
'stdev=568.81, samples=14\n'
' lat (nsec) : 1000=0.03%\n'
' lat (usec) : 2=0.01%, '
'4=3.78%, 10=23.01%, 20=50.43%, '
'50=22.68%\n'
' lat (usec) : 100=0.07%, '
'250=0.01%\n'
' cpu : usr=39.01%, '
'sys=60.96%, ctx=27, majf=0, '
'minf=0\n'
' IO depths : 1=0.4%, '
'2=26.8%, 4=58.0%, 8=14.8%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=87.5%, 8=12.5%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2054717,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=143MiB/s (150MB/s), '
'143MiB/s-143MiB/s '
'(150MB/s-150MB/s), io=1003MiB '
'(1052MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.output'),
'state': {'begin': 1666745656.7511828,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8129782676696777,
'end': 1666745657.564161,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.state')},
'cmd_382': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.output'),
'state': {'begin': 1666745659.6146078,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7873172760009766,
'end': 1666745660.401925,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.state')},
'cmd_383': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.output'),
'state': {'begin': 1666745660.4042623,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.560972213745117,
'end': 1666745671.9652345,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.state')},
'cmd_384': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752761,\n'
' "timestamp_ms" : '
'1666752761025,\n'
' "time" : "Wed Oct 26 04:52:41 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1073317888,\n'
' "io_kbytes" : 1048162,\n'
' "bw_bytes" : 153309225,\n'
' "bw" : 149716,\n'
' "iops" : 299431.366948,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2096319,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1593,\n'
' "max" : 80861,\n'
' "mean" : 2771.613320,\n'
' "stddev" : '
'938.858044,\n'
' "N" : 2096320\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 732,\n'
' "max" : 127409,\n'
' "mean" : '
'14376.397846,\n'
' "stddev" : '
'6697.479279,\n'
' "N" : 2096324,\n'
' "percentile" : {\n'
' "1.000000" : 3792,\n'
' "5.000000" : 3888,\n'
' "10.000000" : 5152,\n'
' "20.000000" : 7200,\n'
' "30.000000" : '
'10048,\n'
' "40.000000" : '
'11840,\n'
' "50.000000" : '
'14528,\n'
' "60.000000" : '
'16768,\n'
' "70.000000" : '
'18304,\n'
' "80.000000" : '
'19840,\n'
' "90.000000" : '
'23680,\n'
' "95.000000" : '
'24192,\n'
' "99.000000" : '
'27264,\n'
' "99.500000" : '
'29824,\n'
' "99.900000" : '
'41728,\n'
' "99.950000" : '
'56576,\n'
' "99.990000" : 70144\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6502,\n'
' "max" : 129052,\n'
' "mean" : '
'17148.011537,\n'
' "stddev" : '
'6676.946354,\n'
' "N" : 2096324\n'
' },\n'
' "bw_min" : 149307,\n'
' "bw_max" : 150245,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'149750.857143,\n'
' "bw_dev" : 298.817891,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 298614,\n'
' "iops_max" : 300490,\n'
' "iops_mean" : '
'299501.714286,\n'
' "iops_stddev" : '
'597.243338,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.985714,\n'
' "sys_cpu" : 59.985714,\n'
' "ctx" : 25,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.448739,\n'
' "2" : 26.840428,\n'
' "4" : 57.885036,\n'
' "8" : 14.825797,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 87.522870,\n'
' "8" : 12.477130,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.039975\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 6.707996,\n'
' "10" : 23.033327,\n'
' "20" : 51.503469,\n'
' "50" : 18.641581,\n'
' "100" : 0.071936,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=144253: Wed '
'Oct 26 04:52:41 2022\n'
' read: IOPS=299k, BW=146MiB/s '
'(153MB/s)(1024MiB/7001msec)\n'
' slat (nsec): min=1593, '
'max=80861, avg=2771.61, '
'stdev=938.86\n'
' clat (nsec): min=732, '
'max=127409, avg=14376.40, '
'stdev=6697.48\n'
' lat (usec): min=6, max=129, '
'avg=17.15, stdev= 6.68\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3792], '
'5.00th=[ 3888], 10.00th=[ 5152], '
'20.00th=[ 7200],\n'
' | 30.00th=[10048], '
'40.00th=[11840], '
'50.00th=[14528], '
'60.00th=[16768],\n'
' | 70.00th=[18304], '
'80.00th=[19840], '
'90.00th=[23680], '
'95.00th=[24192],\n'
' | 99.00th=[27264], '
'99.50th=[29824], '
'99.90th=[41728], '
'99.95th=[56576],\n'
' | 99.99th=[70144]\n'
' bw ( KiB/s): min=149307, '
'max=150245, per=100.00%, '
'avg=149750.86, stdev=298.82, '
'samples=14\n'
' iops : min=298614, '
'max=300490, avg=299501.71, '
'stdev=597.24, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.04%\n'
' lat (usec) : 2=0.01%, '
'4=6.71%, 10=23.03%, 20=51.50%, '
'50=18.64%\n'
' lat (usec) : 100=0.07%, '
'250=0.01%\n'
' cpu : usr=39.99%, '
'sys=59.99%, ctx=25, majf=0, '
'minf=0\n'
' IO depths : 1=0.4%, '
'2=26.8%, 4=57.9%, 8=14.8%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=87.5%, 8=12.5%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2096319,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=146MiB/s (153MB/s), '
'146MiB/s-146MiB/s '
'(153MB/s-153MB/s), io=1024MiB '
'(1073MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.output'),
'state': {'begin': 1666745671.9682,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7915475368499756,
'end': 1666745672.7597475,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.state')},
'cmd_385': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.output'),
'state': {'begin': 1666745674.8022122,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7958424091339111,
'end': 1666745675.5980546,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.state')},
'cmd_386': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.output'),
'state': {'begin': 1666745675.6002083,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.533339262008667,
'end': 1666745687.1335475,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.state')},
'cmd_387': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752776,\n'
' "timestamp_ms" : '
'1666752776202,\n'
' "time" : "Wed Oct 26 04:52:56 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1060783104,\n'
' "io_kbytes" : 1035921,\n'
' "bw_bytes" : 151518797,\n'
' "bw" : 147967,\n'
' "iops" : 295934.152264,\n'
' "runtime" : 7001,\n'
' "total_ios" : 2071835,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1623,\n'
' "max" : 49042,\n'
' "mean" : 2860.583254,\n'
' "stddev" : '
'1041.203387,\n'
' "N" : 2071836\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 762,\n'
' "max" : 127149,\n'
' "mean" : '
'14498.881318,\n'
' "stddev" : '
'6774.396456,\n'
' "N" : 2071842,\n'
' "percentile" : {\n'
' "1.000000" : 3856,\n'
' "5.000000" : 3984,\n'
' "10.000000" : 5216,\n'
' "20.000000" : 8256,\n'
' "30.000000" : '
'10176,\n'
' "40.000000" : '
'11968,\n'
' "50.000000" : '
'14528,\n'
' "60.000000" : '
'17280,\n'
' "70.000000" : '
'18560,\n'
' "80.000000" : '
'19840,\n'
' "90.000000" : '
'23936,\n'
' "95.000000" : '
'24448,\n'
' "99.000000" : '
'27520,\n'
' "99.500000" : '
'30592,\n'
' "99.900000" : '
'46336,\n'
' "99.950000" : '
'58624,\n'
' "99.990000" : 72192\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6472,\n'
' "max" : 128822,\n'
' "mean" : '
'17359.464180,\n'
' "stddev" : '
'6757.687788,\n'
' "N" : 2071842\n'
' },\n'
' "bw_min" : 147584,\n'
' "bw_max" : 148654,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'148008.500000,\n'
' "bw_dev" : 324.912710,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 295168,\n'
' "iops_max" : 297308,\n'
' "iops_mean" : '
'296017.000000,\n'
' "iops_stddev" : '
'650.166251,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.242857,\n'
' "sys_cpu" : 60.728571,\n'
' "ctx" : 24,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.416249,\n'
' "2" : 26.862419,\n'
' "4" : 57.902777,\n'
' "8" : 14.818555,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 87.526588,\n'
' "8" : 12.473412,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.035331\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 5.719471,\n'
' "10" : 21.807528,\n'
' "20" : 52.966718,\n'
' "50" : 19.386679,\n'
' "100" : 0.083260,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=144302: Wed '
'Oct 26 04:52:56 2022\n'
' read: IOPS=296k, BW=144MiB/s '
'(152MB/s)(1012MiB/7001msec)\n'
' slat (nsec): min=1623, '
'max=49042, avg=2860.58, '
'stdev=1041.20\n'
' clat (nsec): min=762, '
'max=127149, avg=14498.88, '
'stdev=6774.40\n'
' lat (usec): min=6, max=128, '
'avg=17.36, stdev= 6.76\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 3856], '
'5.00th=[ 3984], 10.00th=[ 5216], '
'20.00th=[ 8256],\n'
' | 30.00th=[10176], '
'40.00th=[11968], '
'50.00th=[14528], '
'60.00th=[17280],\n'
' | 70.00th=[18560], '
'80.00th=[19840], '
'90.00th=[23936], '
'95.00th=[24448],\n'
' | 99.00th=[27520], '
'99.50th=[30592], '
'99.90th=[46336], '
'99.95th=[58624],\n'
' | 99.99th=[72192]\n'
' bw ( KiB/s): min=147584, '
'max=148654, per=100.00%, '
'avg=148008.50, stdev=324.91, '
'samples=14\n'
' iops : min=295168, '
'max=297308, avg=296017.00, '
'stdev=650.17, samples=14\n'
' lat (nsec) : 1000=0.04%\n'
' lat (usec) : 2=0.01%, '
'4=5.72%, 10=21.81%, 20=52.97%, '
'50=19.39%\n'
' lat (usec) : 100=0.08%, '
'250=0.01%\n'
' cpu : usr=39.24%, '
'sys=60.73%, ctx=24, majf=0, '
'minf=0\n'
' IO depths : 1=0.4%, '
'2=26.9%, 4=57.9%, 8=14.8%, '
'16=0.0%, 32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=87.5%, 8=12.5%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=2071835,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=144MiB/s (152MB/s), '
'144MiB/s-144MiB/s '
'(152MB/s-152MB/s), io=1012MiB '
'(1061MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.output'),
'state': {'begin': 1666745687.1367385,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7910754680633545,
'end': 1666745687.927814,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.state')},
'cmd_388': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.output'),
'state': {'begin': 1666745688.926372,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7844598293304443,
'end': 1666745689.7108319,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.state')},
'cmd_389': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.output'),
'state': {'begin': 1666745689.7130206,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.091380596160889,
'end': 1666745700.8044012,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.state')},
'cmd_39': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751109,\n'
' "timestamp_ms" : '
'1666751109430,\n'
' "time" : "Wed Oct 26 04:25:09 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 554295808,\n'
' "io_kbytes" : 541304,\n'
' "bw_bytes" : 79173804,\n'
' "bw" : 77318,\n'
' "iops" : 154636.337666,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1082609,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 160,\n'
' "max" : 11602,\n'
' "mean" : 177.716369,\n'
' "stddev" : 44.536553,\n'
' "N" : 1082609\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5681,\n'
' "max" : 127730,\n'
' "mean" : 6024.580633,\n'
' "stddev" : '
'1112.814196,\n'
' "N" : 1082609,\n'
' "percentile" : {\n'
' "1.000000" : 5728,\n'
' "5.000000" : 5792,\n'
' "10.000000" : 5792,\n'
' "20.000000" : 5792,\n'
' "30.000000" : 5792,\n'
' "40.000000" : 5856,\n'
' "50.000000" : 5856,\n'
' "60.000000" : 5920,\n'
' "70.000000" : 5920,\n'
' "80.000000" : 5920,\n'
' "90.000000" : 6048,\n'
' "95.000000" : 6112,\n'
' "99.000000" : 11328,\n'
' "99.500000" : 12608,\n'
' "99.900000" : 17024,\n'
' "99.950000" : 17024,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5851,\n'
' "max" : 127900,\n'
' "mean" : 6202.297002,\n'
' "stddev" : '
'1114.031677,\n'
' "N" : 1082609\n'
' },\n'
' "bw_min" : 77106,\n'
' "bw_max" : 77614,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'77336.642857,\n'
' "bw_dev" : 146.503141,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 154212,\n'
' "iops_max" : 155228,\n'
' "iops_mean" : '
'154673.285714,\n'
' "iops_stddev" : '
'293.006282,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 51.814286,\n'
' "sys_cpu" : 48.157143,\n'
' "ctx" : 21,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.603836,\n'
' "20" : 1.381108,\n'
' "50" : 0.010000,\n'
' "100" : 0.012378,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=139184: Wed '
'Oct 26 04:25:09 2022\n'
' read: IOPS=155k, BW=75.5MiB/s '
'(79.2MB/s)(529MiB/7001msec)\n'
' slat (nsec): min=160, '
'max=11602, avg=177.72, '
'stdev=44.54\n'
' clat (usec): min=5, max=127, '
'avg= 6.02, stdev= 1.11\n'
' lat (usec): min=5, max=127, '
'avg= 6.20, stdev= 1.11\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5728], 5.00th=[ '
'5792], 10.00th=[ 5792], 20.00th=[ '
'5792],\n'
' | 30.00th=[ 5792], 40.00th=[ '
'5856], 50.00th=[ 5856], 60.00th=[ '
'5920],\n'
' | 70.00th=[ 5920], 80.00th=[ '
'5920], 90.00th=[ 6048], 95.00th=[ '
'6112],\n'
' | 99.00th=[11328], '
'99.50th=[12608], 99.90th=[17024], '
'99.95th=[17024],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=77106, '
'max=77614, per=100.00%, '
'avg=77336.64, stdev=146.50, '
'samples=14\n'
' iops : min=154212, '
'max=155228, avg=154673.29, '
'stdev=293.01, samples=14\n'
' lat (usec) : 10=98.60%, '
'20=1.38%, 50=0.01%, 100=0.01%, '
'250=0.01%\n'
' cpu : usr=51.81%, '
'sys=48.16%, ctx=21, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1082609,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=75.5MiB/s (79.2MB/s), '
'75.5MiB/s-75.5MiB/s '
'(79.2MB/s-79.2MB/s), io=529MiB '
'(554MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.output'),
'state': {'begin': 1666744020.303161,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt',
'cwd': 'None',
'elapsed': 0.8060753345489502,
'end': 1666744021.1092362,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.state')},
'cmd_390': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752789,\n'
' "timestamp_ms" : '
'1666752789979,\n'
' "time" : "Wed Oct 26 04:53:09 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1014642688,\n'
' "io_kbytes" : 990862,\n'
' "bw_bytes" : 144928251,\n'
' "bw" : 141531,\n'
' "iops" : 283061.848307,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1981716,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1472,\n'
' "max" : 32762,\n'
' "mean" : 2358.838814,\n'
' "stddev" : '
'1033.707278,\n'
' "N" : 1981717\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 6823,\n'
' "max" : 153127,\n'
' "mean" : '
'25221.540123,\n'
' "stddev" : '
'3166.343719,\n'
' "N" : 1981723,\n'
' "percentile" : {\n'
' "1.000000" : 18304,\n'
' "5.000000" : 23168,\n'
' "10.000000" : '
'23424,\n'
' "20.000000" : '
'23424,\n'
' "30.000000" : '
'23680,\n'
' "40.000000" : '
'23680,\n'
' "50.000000" : '
'24960,\n'
' "60.000000" : '
'25984,\n'
' "70.000000" : '
'26240,\n'
' "80.000000" : '
'26240,\n'
' "90.000000" : '
'26496,\n'
' "95.000000" : '
'29568,\n'
' "99.000000" : '
'38144,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'51968,\n'
' "99.950000" : '
'64768,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 10910,\n'
' "max" : 154660,\n'
' "mean" : '
'27580.380426,\n'
' "stddev" : '
'2971.138538,\n'
' "N" : 1981723\n'
' },\n'
' "bw_min" : 141148,\n'
' "bw_max" : 141868,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'141569.500000,\n'
' "bw_dev" : 202.955755,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 282296,\n'
' "iops_max" : 283736,\n'
' "iops_mean" : '
'283139.142857,\n'
' "iops_stddev" : '
'405.859175,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 36.985714,\n'
' "sys_cpu" : 62.000000,\n'
' "ctx" : 9204,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999975,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 1.889827,\n'
' "50" : 97.980891,\n'
' "100" : 0.126507,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=144341: Wed '
'Oct 26 04:53:09 2022\n'
' read: IOPS=283k, BW=138MiB/s '
'(145MB/s)(968MiB/7001msec)\n'
' slat (nsec): min=1472, '
'max=32762, avg=2358.84, '
'stdev=1033.71\n'
' clat (usec): min=6, max=153, '
'avg=25.22, stdev= 3.17\n'
' lat (usec): min=10, '
'max=154, avg=27.58, stdev= 2.97\n'
' clat percentiles (nsec):\n'
' | 1.00th=[18304], '
'5.00th=[23168], 10.00th=[23424], '
'20.00th=[23424],\n'
' | 30.00th=[23680], '
'40.00th=[23680], '
'50.00th=[24960], '
'60.00th=[25984],\n'
' | 70.00th=[26240], '
'80.00th=[26240], '
'90.00th=[26496], '
'95.00th=[29568],\n'
' | 99.00th=[38144], '
'99.50th=[42752], '
'99.90th=[51968], '
'99.95th=[64768],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=141148, '
'max=141868, per=100.00%, '
'avg=141569.50, stdev=202.96, '
'samples=14\n'
' iops : min=282296, '
'max=283736, avg=283139.14, '
'stdev=405.86, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=1.89%, 50=97.98%, 100=0.13%, '
'250=0.01%\n'
' cpu : usr=36.99%, '
'sys=62.00%, ctx=9204, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1981716,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=138MiB/s (145MB/s), '
'138MiB/s-138MiB/s '
'(145MB/s-145MB/s), io=968MiB '
'(1015MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.output'),
'state': {'begin': 1666745700.8076355,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.8106334209442139,
'end': 1666745701.618269,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.state')},
'cmd_391': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.output'),
'state': {'begin': 1666745702.6029127,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7850196361541748,
'end': 1666745703.3879323,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.state')},
'cmd_392': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.output'),
'state': {'begin': 1666745703.3901885,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.085580110549927,
'end': 1666745714.4757686,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.state')},
'cmd_393': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752803,\n'
' "timestamp_ms" : '
'1666752803651,\n'
' "time" : "Wed Oct 26 04:53:23 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1012346368,\n'
' "io_kbytes" : 988619,\n'
' "bw_bytes" : 144600252,\n'
' "bw" : 141211,\n'
' "iops" : 282421.368376,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1977232,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1482,\n'
' "max" : 54362,\n'
' "mean" : 2372.692545,\n'
' "stddev" : '
'1039.385184,\n'
' "N" : 1977232\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4679,\n'
' "max" : 146735,\n'
' "mean" : '
'25274.840964,\n'
' "stddev" : '
'3228.285563,\n'
' "N" : 1977239,\n'
' "percentile" : {\n'
' "1.000000" : 18304,\n'
' "5.000000" : 23168,\n'
' "10.000000" : '
'23424,\n'
' "20.000000" : '
'23424,\n'
' "30.000000" : '
'23680,\n'
' "40.000000" : '
'23936,\n'
' "50.000000" : '
'24960,\n'
' "60.000000" : '
'25984,\n'
' "70.000000" : '
'26240,\n'
' "80.000000" : '
'26240,\n'
' "90.000000" : '
'26496,\n'
' "95.000000" : '
'30080,\n'
' "99.000000" : '
'38656,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'53504,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 79360\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6211,\n'
' "max" : 151143,\n'
' "mean" : '
'27647.531768,\n'
' "stddev" : '
'3048.297465,\n'
' "N" : 1977239\n'
' },\n'
' "bw_min" : 140956,\n'
' "bw_max" : 141621,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'141248.214286,\n'
' "bw_dev" : 159.472293,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 281912,\n'
' "iops_max" : 283242,\n'
' "iops_mean" : '
'282496.500000,\n'
' "iops_stddev" : '
'318.744351,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 38.000000,\n'
' "sys_cpu" : 60.957143,\n'
' "ctx" : 9034,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999975,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 1.892090,\n'
' "50" : 97.959976,\n'
' "100" : 0.144242,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=144380: Wed '
'Oct 26 04:53:23 2022\n'
' read: IOPS=282k, BW=138MiB/s '
'(145MB/s)(965MiB/7001msec)\n'
' slat (nsec): min=1482, '
'max=54362, avg=2372.69, '
'stdev=1039.39\n'
' clat (usec): min=4, max=146, '
'avg=25.27, stdev= 3.23\n'
' lat (usec): min=6, max=151, '
'avg=27.65, stdev= 3.05\n'
' clat percentiles (nsec):\n'
' | 1.00th=[18304], '
'5.00th=[23168], 10.00th=[23424], '
'20.00th=[23424],\n'
' | 30.00th=[23680], '
'40.00th=[23936], '
'50.00th=[24960], '
'60.00th=[25984],\n'
' | 70.00th=[26240], '
'80.00th=[26240], '
'90.00th=[26496], '
'95.00th=[30080],\n'
' | 99.00th=[38656], '
'99.50th=[42752], '
'99.90th=[53504], '
'99.95th=[66048],\n'
' | 99.99th=[79360]\n'
' bw ( KiB/s): min=140956, '
'max=141621, per=100.00%, '
'avg=141248.21, stdev=159.47, '
'samples=14\n'
' iops : min=281912, '
'max=283242, avg=282496.50, '
'stdev=318.74, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=1.89%, 50=97.96%, 100=0.14%, '
'250=0.01%\n'
' cpu : usr=38.00%, '
'sys=60.96%, ctx=9034, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1977232,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=138MiB/s (145MB/s), '
'138MiB/s-138MiB/s '
'(145MB/s-145MB/s), io=965MiB '
'(1012MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.output'),
'state': {'begin': 1666745714.4786277,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7916066646575928,
'end': 1666745715.2702343,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.state')},
'cmd_394': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.output'),
'state': {'begin': 1666745716.243132,
'cmd': 'rm '
'/tmp/fio-output_512_8_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7940242290496826,
'end': 1666745717.0371563,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.state')},
'cmd_395': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.output'),
'state': {'begin': 1666745717.0390606,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" '
'--iodepth="8" --bs="512" '
'--direct="1" --thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_8_libaio-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.08297610282898,
'end': 1666745728.1220367,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.state')},
'cmd_396': {'output': 'libaio-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=8\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666752817,\n'
' "timestamp_ms" : '
'1666752817296,\n'
' "time" : "Wed Oct 26 04:53:37 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "8",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : '
'1011662336,\n'
' "io_kbytes" : 987951,\n'
' "bw_bytes" : 144502547,\n'
' "bw" : 141115,\n'
' "iops" : 282230.395658,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1975895,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1482,\n'
' "max" : 25537,\n'
' "mean" : 2298.559704,\n'
' "stddev" : '
'931.379939,\n'
' "N" : 1975896\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 4278,\n'
' "max" : 139502,\n'
' "mean" : '
'25292.891398,\n'
' "stddev" : '
'3159.053006,\n'
' "N" : 1975902,\n'
' "percentile" : {\n'
' "1.000000" : 18304,\n'
' "5.000000" : 23168,\n'
' "10.000000" : '
'23424,\n'
' "20.000000" : '
'23680,\n'
' "30.000000" : '
'23680,\n'
' "40.000000" : '
'23936,\n'
' "50.000000" : '
'24960,\n'
' "60.000000" : '
'25984,\n'
' "70.000000" : '
'26240,\n'
' "80.000000" : '
'26496,\n'
' "90.000000" : '
'26496,\n'
' "95.000000" : '
'29824,\n'
' "99.000000" : '
'38144,\n'
' "99.500000" : '
'42752,\n'
' "99.900000" : '
'52480,\n'
' "99.950000" : '
'66048,\n'
' "99.990000" : 78336\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5811,\n'
' "max" : 141034,\n'
' "mean" : '
'27591.452594,\n'
' "stddev" : '
'2976.337309,\n'
' "N" : 1975902\n'
' },\n'
' "bw_min" : 140763,\n'
' "bw_max" : 141655,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'141148.214286,\n'
' "bw_dev" : 223.325281,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 281526,\n'
' "iops_max" : 283310,\n'
' "iops_mean" : '
'282296.428571,\n'
' "iops_stddev" : '
'446.650563,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 39.814286,\n'
' "sys_cpu" : 59.185714,\n'
' "ctx" : 8781,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 0.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 100.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 99.999975,\n'
' "8" : 0.100000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.010000,\n'
' "20" : 1.898886,\n'
' "50" : 97.964619,\n'
' "100" : 0.134015,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 8,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=144420: Wed '
'Oct 26 04:53:37 2022\n'
' read: IOPS=282k, BW=138MiB/s '
'(145MB/s)(965MiB/7001msec)\n'
' slat (nsec): min=1482, '
'max=25537, avg=2298.56, '
'stdev=931.38\n'
' clat (usec): min=4, max=139, '
'avg=25.29, stdev= 3.16\n'
' lat (usec): min=5, max=141, '
'avg=27.59, stdev= 2.98\n'
' clat percentiles (nsec):\n'
' | 1.00th=[18304], '
'5.00th=[23168], 10.00th=[23424], '
'20.00th=[23680],\n'
' | 30.00th=[23680], '
'40.00th=[23936], '
'50.00th=[24960], '
'60.00th=[25984],\n'
' | 70.00th=[26240], '
'80.00th=[26496], '
'90.00th=[26496], '
'95.00th=[29824],\n'
' | 99.00th=[38144], '
'99.50th=[42752], '
'99.90th=[52480], '
'99.95th=[66048],\n'
' | 99.99th=[78336]\n'
' bw ( KiB/s): min=140763, '
'max=141655, per=100.00%, '
'avg=141148.21, stdev=223.33, '
'samples=14\n'
' iops : min=281526, '
'max=283310, avg=282296.43, '
'stdev=446.65, samples=14\n'
' lat (usec) : 10=0.01%, '
'20=1.90%, 50=97.96%, 100=0.13%, '
'250=0.01%\n'
' cpu : usr=39.81%, '
'sys=59.19%, ctx=8781, majf=0, '
'minf=0\n'
' IO depths : 1=0.0%, 2=0.0%, '
'4=0.0%, 8=100.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.1%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1975895,0,0,0 '
'short=0,0,0,0 dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=8\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=138MiB/s (145MB/s), '
'138MiB/s-138MiB/s '
'(145MB/s-145MB/s), io=965MiB '
'(1012MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.output'),
'state': {'begin': 1666745728.1251886,
'cmd': 'cat '
'/tmp/fio-output_512_8_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7885286808013916,
'end': 1666745728.9137173,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.state')},
'cmd_40': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.output'),
'state': {'begin': 1666744023.1586106,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7830297946929932,
'end': 1666744023.9416404,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.state')},
'cmd_41': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.output'),
'state': {'begin': 1666744023.943622,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.344534397125244,
'end': 1666744035.2881565,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.state')},
'cmd_42': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751124,\n'
' "timestamp_ms" : '
'1666751124416,\n'
' "time" : "Wed Oct 26 04:25:24 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 554997760,\n'
' "io_kbytes" : 541990,\n'
' "bw_bytes" : 79285394,\n'
' "bw" : 77427,\n'
' "iops" : 154854.285714,\n'
' "runtime" : 7000,\n'
' "total_ios" : 1083980,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 160,\n'
' "max" : 12163,\n'
' "mean" : 182.966398,\n'
' "stddev" : 38.361146,\n'
' "N" : 1083980\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5690,\n'
' "max" : 79259,\n'
' "mean" : 6045.423559,\n'
' "stddev" : '
'1095.010971,\n'
' "N" : 1083980,\n'
' "percentile" : {\n'
' "1.000000" : 5792,\n'
' "5.000000" : 5792,\n'
' "10.000000" : 5792,\n'
' "20.000000" : 5856,\n'
' "30.000000" : 5856,\n'
' "40.000000" : 5856,\n'
' "50.000000" : 5920,\n'
' "60.000000" : 5920,\n'
' "70.000000" : 5920,\n'
' "80.000000" : 5984,\n'
' "90.000000" : 6048,\n'
' "95.000000" : 6176,\n'
' "99.000000" : 11456,\n'
' "99.500000" : 12096,\n'
' "99.900000" : 17024,\n'
' "99.950000" : 17024,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5871,\n'
' "max" : 79429,\n'
' "mean" : 6228.389957,\n'
' "stddev" : '
'1095.955735,\n'
' "N" : 1083980\n'
' },\n'
' "bw_min" : 77263,\n'
' "bw_max" : 77571,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'77436.857143,\n'
' "bw_dev" : 79.987362,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 154526,\n'
' "iops_max" : 155142,\n'
' "iops_mean" : '
'154873.785714,\n'
' "iops_stddev" : '
'160.033736,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 48.900000,\n'
' "sys_cpu" : 51.071429,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.581247,\n'
' "20" : 1.403808,\n'
' "50" : 0.010000,\n'
' "100" : 0.013100,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=139233: Wed '
'Oct 26 04:25:24 2022\n'
' read: IOPS=155k, BW=75.6MiB/s '
'(79.3MB/s)(529MiB/7000msec)\n'
' slat (nsec): min=160, '
'max=12163, avg=182.97, '
'stdev=38.36\n'
' clat (nsec): min=5690, '
'max=79259, avg=6045.42, '
'stdev=1095.01\n'
' lat (nsec): min=5871, '
'max=79429, avg=6228.39, '
'stdev=1095.96\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5792], 5.00th=[ '
'5792], 10.00th=[ 5792], 20.00th=[ '
'5856],\n'
' | 30.00th=[ 5856], 40.00th=[ '
'5856], 50.00th=[ 5920], 60.00th=[ '
'5920],\n'
' | 70.00th=[ 5920], 80.00th=[ '
'5984], 90.00th=[ 6048], 95.00th=[ '
'6176],\n'
' | 99.00th=[11456], '
'99.50th=[12096], 99.90th=[17024], '
'99.95th=[17024],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=77263, '
'max=77571, per=100.00%, '
'avg=77436.86, stdev=79.99, '
'samples=14\n'
' iops : min=154526, '
'max=155142, avg=154873.79, '
'stdev=160.03, samples=14\n'
' lat (usec) : 10=98.58%, '
'20=1.40%, 50=0.01%, 100=0.01%\n'
' cpu : usr=48.90%, '
'sys=51.07%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1083980,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=75.6MiB/s (79.3MB/s), '
'75.6MiB/s-75.6MiB/s '
'(79.3MB/s-79.3MB/s), io=529MiB '
'(555MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.output'),
'state': {'begin': 1666744035.290713,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt',
'cwd': 'None',
'elapsed': 0.7952075004577637,
'end': 1666744036.0859206,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.state')},
'cmd_43': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.output'),
'state': {'begin': 1666744038.146734,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7901654243469238,
'end': 1666744038.9368994,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.state')},
'cmd_44': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.output'),
'state': {'begin': 1666744038.9389307,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_uring" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.368819952011108,
'end': 1666744050.3077507,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.state')},
'cmd_45': {'output': 'io_uring-bdev_uring: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751139,\n'
' "timestamp_ms" : '
'1666751139437,\n'
' "time" : "Wed Oct 26 04:25:39 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_uring",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_uring",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_uring_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 556163072,\n'
' "io_kbytes" : 543128,\n'
' "bw_bytes" : 79440518,\n'
' "bw" : 77578,\n'
' "iops" : 155157.263248,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1086256,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 160,\n'
' "max" : 11702,\n'
' "mean" : 182.538771,\n'
' "stddev" : 44.114911,\n'
' "N" : 1086256\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 5681,\n'
' "max" : 114124,\n'
' "mean" : 6027.226244,\n'
' "stddev" : '
'1126.443798,\n'
' "N" : 1086256,\n'
' "percentile" : {\n'
' "1.000000" : 5728,\n'
' "5.000000" : 5792,\n'
' "10.000000" : 5792,\n'
' "20.000000" : 5792,\n'
' "30.000000" : 5792,\n'
' "40.000000" : 5856,\n'
' "50.000000" : 5856,\n'
' "60.000000" : 5920,\n'
' "70.000000" : 5920,\n'
' "80.000000" : 5920,\n'
' "90.000000" : 6048,\n'
' "95.000000" : 6112,\n'
' "99.000000" : 11456,\n'
' "99.500000" : 12736,\n'
' "99.900000" : 17024,\n'
' "99.950000" : 17024,\n'
' "99.990000" : 58624\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5861,\n'
' "max" : 114305,\n'
' "mean" : 6209.765015,\n'
' "stddev" : '
'1127.684770,\n'
' "N" : 1086256\n'
' },\n'
' "bw_min" : 77442,\n'
' "bw_max" : 77945,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'77599.142857,\n'
' "bw_dev" : 130.531374,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 154884,\n'
' "iops_max" : 155890,\n'
' "iops_mean" : '
'155198.357143,\n'
' "iops_stddev" : '
'261.172388,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 47.071429,\n'
' "sys_cpu" : 52.900000,\n'
' "ctx" : 18,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 98.564611,\n'
' "20" : 1.419371,\n'
' "50" : 0.010000,\n'
' "100" : 0.013809,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_uring: (groupid=0, '
'jobs=1): err= 0: pid=139283: Wed '
'Oct 26 04:25:39 2022\n'
' read: IOPS=155k, BW=75.8MiB/s '
'(79.4MB/s)(530MiB/7001msec)\n'
' slat (nsec): min=160, '
'max=11702, avg=182.54, '
'stdev=44.11\n'
' clat (usec): min=5, max=114, '
'avg= 6.03, stdev= 1.13\n'
' lat (usec): min=5, max=114, '
'avg= 6.21, stdev= 1.13\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5728], 5.00th=[ '
'5792], 10.00th=[ 5792], 20.00th=[ '
'5792],\n'
' | 30.00th=[ 5792], 40.00th=[ '
'5856], 50.00th=[ 5856], 60.00th=[ '
'5920],\n'
' | 70.00th=[ 5920], 80.00th=[ '
'5920], 90.00th=[ 6048], 95.00th=[ '
'6112],\n'
' | 99.00th=[11456], '
'99.50th=[12736], 99.90th=[17024], '
'99.95th=[17024],\n'
' | 99.99th=[58624]\n'
' bw ( KiB/s): min=77442, '
'max=77945, per=100.00%, '
'avg=77599.14, stdev=130.53, '
'samples=14\n'
' iops : min=154884, '
'max=155890, avg=155198.36, '
'stdev=261.17, samples=14\n'
' lat (usec) : 10=98.56%, '
'20=1.42%, 50=0.01%, 100=0.01%, '
'250=0.01%\n'
' cpu : usr=47.07%, '
'sys=52.90%, ctx=18, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1086256,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=75.8MiB/s (79.4MB/s), '
'75.8MiB/s-75.8MiB/s '
'(79.4MB/s-79.4MB/s), io=530MiB '
'(556MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.output'),
'state': {'begin': 1666744050.3104663,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt',
'cwd': 'None',
'elapsed': 0.7914083003997803,
'end': 1666744051.1018746,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.state')},
'cmd_46': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.output'),
'state': {'begin': 1666744053.1546078,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7848234176635742,
'end': 1666744053.9394312,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.state')},
'cmd_47': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.output'),
'state': {'begin': 1666744053.9416974,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.334166526794434,
'end': 1666744065.275864,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.state')},
'cmd_48': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751154,\n'
' "timestamp_ms" : '
'1666751154404,\n'
' "time" : "Wed Oct 26 04:25:54 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 579297280,\n'
' "io_kbytes" : 565720,\n'
' "bw_bytes" : 82744933,\n'
' "bw" : 80805,\n'
' "iops" : 161611.055563,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1131439,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1573,\n'
' "max" : 22613,\n'
' "mean" : 1629.937999,\n'
' "stddev" : 110.064556,\n'
' "N" : 1131440\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 231,\n'
' "max" : 100949,\n'
' "mean" : 4328.043267,\n'
' "stddev" : '
'1152.033943,\n'
' "N" : 1131439,\n'
' "percentile" : {\n'
' "1.000000" : 4048,\n'
' "5.000000" : 4080,\n'
' "10.000000" : 4080,\n'
' "20.000000" : 4128,\n'
' "30.000000" : 4128,\n'
' "40.000000" : 4128,\n'
' "50.000000" : 4192,\n'
' "60.000000" : 4192,\n'
' "70.000000" : 4256,\n'
' "80.000000" : 4256,\n'
' "90.000000" : 4320,\n'
' "95.000000" : 4448,\n'
' "99.000000" : 10048,\n'
' "99.500000" : 10688,\n'
' "99.900000" : 15424,\n'
' "99.950000" : 15552,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5620,\n'
' "max" : 102592,\n'
' "mean" : 5957.981281,\n'
' "stddev" : '
'1158.044008,\n'
' "N" : 1131439\n'
' },\n'
' "bw_min" : 80689,\n'
' "bw_max" : 80965,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'80824.285714,\n'
' "bw_dev" : 84.986360,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 161378,\n'
' "iops_max" : 161930,\n'
' "iops_mean" : '
'161648.642857,\n'
' "iops_stddev" : '
'169.934651,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.557143,\n'
' "sys_cpu" : 89.414286,\n'
' "ctx" : 24,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.957080,\n'
' "20" : 1.019763,\n'
' "50" : 0.010000,\n'
' "100" : 0.014495,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=139333: Wed '
'Oct 26 04:25:54 2022\n'
' read: IOPS=162k, BW=78.9MiB/s '
'(82.7MB/s)(552MiB/7001msec)\n'
' slat (nsec): min=1573, '
'max=22613, avg=1629.94, '
'stdev=110.06\n'
' clat (nsec): min=231, '
'max=100949, avg=4328.04, '
'stdev=1152.03\n'
' lat (usec): min=5, max=102, '
'avg= 5.96, stdev= 1.16\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4048], 5.00th=[ '
'4080], 10.00th=[ 4080], 20.00th=[ '
'4128],\n'
' | 30.00th=[ 4128], 40.00th=[ '
'4128], 50.00th=[ 4192], 60.00th=[ '
'4192],\n'
' | 70.00th=[ 4256], 80.00th=[ '
'4256], 90.00th=[ 4320], 95.00th=[ '
'4448],\n'
' | 99.00th=[10048], '
'99.50th=[10688], 99.90th=[15424], '
'99.95th=[15552],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=80689, '
'max=80965, per=100.00%, '
'avg=80824.29, stdev=84.99, '
'samples=14\n'
' iops : min=161378, '
'max=161930, avg=161648.64, '
'stdev=169.93, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.96%, 20=1.02%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%, '
'250=0.01%\n'
' cpu : usr=10.56%, '
'sys=89.41%, ctx=24, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1131439,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=78.9MiB/s (82.7MB/s), '
'78.9MiB/s-78.9MiB/s '
'(82.7MB/s-82.7MB/s), io=552MiB '
'(579MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.output'),
'state': {'begin': 1666744065.2787638,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.797447681427002,
'end': 1666744066.0762115,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.state')},
'cmd_49': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.output'),
'state': {'begin': 1666744068.102742,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.789142370223999,
'end': 1666744068.8918843,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.state')},
'cmd_50': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.output'),
'state': {'begin': 1666744068.8940985,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.456218719482422,
'end': 1666744080.3503172,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.state')},
'cmd_51': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751169,\n'
' "timestamp_ms" : '
'1666751169479,\n'
' "time" : "Wed Oct 26 04:26:09 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 578703872,\n'
' "io_kbytes" : 565140,\n'
' "bw_bytes" : 82671981,\n'
' "bw" : 80734,\n'
' "iops" : 161468.714286,\n'
' "runtime" : 7000,\n'
' "total_ios" : 1130281,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1563,\n'
' "max" : 15979,\n'
' "mean" : 1627.234484,\n'
' "stddev" : 111.403997,\n'
' "N" : 1130281\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 230,\n'
' "max" : 75993,\n'
' "mean" : 4333.185862,\n'
' "stddev" : '
'1142.444879,\n'
' "N" : 1130281,\n'
' "percentile" : {\n'
' "1.000000" : 4080,\n'
' "5.000000" : 4080,\n'
' "10.000000" : 4128,\n'
' "20.000000" : 4128,\n'
' "30.000000" : 4128,\n'
' "40.000000" : 4192,\n'
' "50.000000" : 4192,\n'
' "60.000000" : 4192,\n'
' "70.000000" : 4256,\n'
' "80.000000" : 4256,\n'
' "90.000000" : 4320,\n'
' "95.000000" : 4448,\n'
' "99.000000" : 10048,\n'
' "99.500000" : 11200,\n'
' "99.900000" : 15424,\n'
' "99.950000" : 15552,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5630,\n'
' "max" : 77626,\n'
' "mean" : 5960.420346,\n'
' "stddev" : '
'1148.508228,\n'
' "N" : 1130281\n'
' },\n'
' "bw_min" : 80670,\n'
' "bw_max" : 80870,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'80741.785714,\n'
' "bw_dev" : 54.574125,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 161340,\n'
' "iops_max" : 161740,\n'
' "iops_mean" : '
'161483.571429,\n'
' "iops_stddev" : '
'109.148251,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 11.014286,\n'
' "sys_cpu" : 88.957143,\n'
' "ctx" : 20,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.965390,\n'
' "20" : 1.013199,\n'
' "50" : 0.010000,\n'
' "100" : 0.012740,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=139382: Wed '
'Oct 26 04:26:09 2022\n'
' read: IOPS=161k, BW=78.8MiB/s '
'(82.7MB/s)(552MiB/7000msec)\n'
' slat (nsec): min=1563, '
'max=15979, avg=1627.23, '
'stdev=111.40\n'
' clat (nsec): min=230, '
'max=75993, avg=4333.19, '
'stdev=1142.44\n'
' lat (nsec): min=5630, '
'max=77626, avg=5960.42, '
'stdev=1148.51\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4080], 5.00th=[ '
'4080], 10.00th=[ 4128], 20.00th=[ '
'4128],\n'
' | 30.00th=[ 4128], 40.00th=[ '
'4192], 50.00th=[ 4192], 60.00th=[ '
'4192],\n'
' | 70.00th=[ 4256], 80.00th=[ '
'4256], 90.00th=[ 4320], 95.00th=[ '
'4448],\n'
' | 99.00th=[10048], '
'99.50th=[11200], 99.90th=[15424], '
'99.95th=[15552],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=80670, '
'max=80870, per=100.00%, '
'avg=80741.79, stdev=54.57, '
'samples=14\n'
' iops : min=161340, '
'max=161740, avg=161483.57, '
'stdev=109.15, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.97%, 20=1.01%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=11.01%, '
'sys=88.96%, ctx=20, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1130281,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=78.8MiB/s (82.7MB/s), '
'78.8MiB/s-78.8MiB/s '
'(82.7MB/s-82.7MB/s), io=552MiB '
'(579MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.output'),
'state': {'begin': 1666744080.3534555,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7853691577911377,
'end': 1666744081.1388247,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.state')},
'cmd_52': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.output'),
'state': {'begin': 1666744083.15069,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7832794189453125,
'end': 1666744083.9339695,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.state')},
'cmd_53': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.output'),
'state': {'begin': 1666744083.9360201,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.43041706085205,
'end': 1666744095.3664372,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.state')},
'cmd_54': {'output': 'io_uring-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751184,\n'
' "timestamp_ms" : '
'1666751184500,\n'
' "time" : "Wed Oct 26 04:26:24 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 576747520,\n'
' "io_kbytes" : 563230,\n'
' "bw_bytes" : 82380734,\n'
' "bw" : 80449,\n'
' "iops" : 160899.871447,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1126460,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1563,\n'
' "max" : 15619,\n'
' "mean" : 1623.509082,\n'
' "stddev" : 104.723698,\n'
' "N" : 1126460\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 241,\n'
' "max" : 77606,\n'
' "mean" : 4359.351258,\n'
' "stddev" : '
'1122.867630,\n'
' "N" : 1126460,\n'
' "percentile" : {\n'
' "1.000000" : 4128,\n'
' "5.000000" : 4128,\n'
' "10.000000" : 4128,\n'
' "20.000000" : 4128,\n'
' "30.000000" : 4128,\n'
' "40.000000" : 4192,\n'
' "50.000000" : 4192,\n'
' "60.000000" : 4256,\n'
' "70.000000" : 4256,\n'
' "80.000000" : 4320,\n'
' "90.000000" : 4384,\n'
' "95.000000" : 4448,\n'
' "99.000000" : 10048,\n'
' "99.500000" : 10688,\n'
' "99.900000" : 15424,\n'
' "99.950000" : 15552,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5650,\n'
' "max" : 79209,\n'
' "mean" : 5982.860339,\n'
' "stddev" : '
'1128.826196,\n'
' "N" : 1126460\n'
' },\n'
' "bw_min" : 80316,\n'
' "bw_max" : 80630,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'80486.615385,\n'
' "bw_dev" : 102.524419,\n'
' "bw_samples" : 13,\n'
' "iops_min" : 160632,\n'
' "iops_max" : 161260,\n'
' "iops_mean" : '
'160973.230769,\n'
' "iops_stddev" : '
'205.048837,\n'
' "iops_samples" : 13\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 10.328571,\n'
' "sys_cpu" : 89.642857,\n'
' "ctx" : 21,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.971468,\n'
' "20" : 1.008291,\n'
' "50" : 0.010000,\n'
' "100" : 0.013050,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=139431: Wed '
'Oct 26 04:26:24 2022\n'
' read: IOPS=161k, BW=78.6MiB/s '
'(82.4MB/s)(550MiB/7001msec)\n'
' slat (nsec): min=1563, '
'max=15619, avg=1623.51, '
'stdev=104.72\n'
' clat (nsec): min=241, '
'max=77606, avg=4359.35, '
'stdev=1122.87\n'
' lat (nsec): min=5650, '
'max=79209, avg=5982.86, '
'stdev=1128.83\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4128], 5.00th=[ '
'4128], 10.00th=[ 4128], 20.00th=[ '
'4128],\n'
' | 30.00th=[ 4128], 40.00th=[ '
'4192], 50.00th=[ 4192], 60.00th=[ '
'4256],\n'
' | 70.00th=[ 4256], 80.00th=[ '
'4320], 90.00th=[ 4384], 95.00th=[ '
'4448],\n'
' | 99.00th=[10048], '
'99.50th=[10688], 99.90th=[15424], '
'99.95th=[15552],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=80316, '
'max=80630, per=100.00%, '
'avg=80486.62, stdev=102.52, '
'samples=13\n'
' iops : min=160632, '
'max=161260, avg=160973.23, '
'stdev=205.05, samples=13\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%, 750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.97%, 20=1.01%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=10.33%, '
'sys=89.64%, ctx=21, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1126460,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=78.6MiB/s (82.4MB/s), '
'78.6MiB/s-78.6MiB/s '
'(82.4MB/s-82.4MB/s), io=550MiB '
'(577MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.output'),
'state': {'begin': 1666744095.3690445,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7837162017822266,
'end': 1666744096.1527607,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.state')},
'cmd_55': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.output'),
'state': {'begin': 1666744097.1452632,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7812321186065674,
'end': 1666744097.9264953,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.state')},
'cmd_56': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.output'),
'state': {'begin': 1666744097.9284656,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.052183866500854,
'end': 1666744108.9806495,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.state')},
'cmd_57': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751198,\n'
' "timestamp_ms" : '
'1666751198179,\n'
' "time" : "Wed Oct 26 04:26:38 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 278675456,\n'
' "io_kbytes" : 272144,\n'
' "bw_bytes" : 39805092,\n'
' "bw" : 38872,\n'
' "iops" : 77744.179403,\n'
' "runtime" : 7001,\n'
' "total_ios" : 544287,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2585,\n'
' "max" : 41207,\n'
' "mean" : 2875.072484,\n'
' "stddev" : 249.042554,\n'
' "N" : 544288\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 301,\n'
' "max" : 155822,\n'
' "mean" : 9423.841889,\n'
' "stddev" : '
'1236.552305,\n'
' "N" : 544287,\n'
' "percentile" : {\n'
' "1.000000" : 9024,\n'
' "5.000000" : 9024,\n'
' "10.000000" : 9152,\n'
' "20.000000" : 9152,\n'
' "30.000000" : 9280,\n'
' "40.000000" : 9280,\n'
' "50.000000" : 9280,\n'
' "60.000000" : 9280,\n'
' "70.000000" : 9280,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 9664,\n'
' "99.000000" : 15168,\n'
' "99.500000" : 15808,\n'
' "99.900000" : 19840,\n'
' "99.950000" : 26752,\n'
' "99.990000" : 55552\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9348,\n'
' "max" : 158678,\n'
' "mean" : 12298.914446,\n'
' "stddev" : '
'1258.957845,\n'
' "N" : 544287\n'
' },\n'
' "bw_min" : 38691,\n'
' "bw_max" : 39063,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'38882.357143,\n'
' "bw_dev" : 110.504860,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 77382,\n'
' "iops_max" : 78126,\n'
' "iops_mean" : '
'77764.714286,\n'
' "iops_stddev" : '
'221.009721,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.557143,\n'
' "sys_cpu" : 47.928571,\n'
' "ctx" : 544144,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.011942,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.766412,\n'
' "20" : 3.138785,\n'
' "50" : 0.065958,\n'
' "100" : 0.016535,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=139470: Wed '
'Oct 26 04:26:38 2022\n'
' read: IOPS=77.7k, BW=38.0MiB/s '
'(39.8MB/s)(266MiB/7001msec)\n'
' slat (nsec): min=2585, '
'max=41207, avg=2875.07, '
'stdev=249.04\n'
' clat (nsec): min=301, '
'max=155822, avg=9423.84, '
'stdev=1236.55\n'
' lat (usec): min=9, max=158, '
'avg=12.30, stdev= 1.26\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9024], 5.00th=[ '
'9024], 10.00th=[ 9152], 20.00th=[ '
'9152],\n'
' | 30.00th=[ 9280], 40.00th=[ '
'9280], 50.00th=[ 9280], 60.00th=[ '
'9280],\n'
' | 70.00th=[ 9280], 80.00th=[ '
'9408], 90.00th=[ 9408], 95.00th=[ '
'9664],\n'
' | 99.00th=[15168], '
'99.50th=[15808], 99.90th=[19840], '
'99.95th=[26752],\n'
' | 99.99th=[55552]\n'
' bw ( KiB/s): min=38691, '
'max=39063, per=100.00%, '
'avg=38882.36, stdev=110.50, '
'samples=14\n'
' iops : min=77382, '
'max=78126, avg=77764.71, '
'stdev=221.01, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 10=96.77%, '
'20=3.14%, 50=0.07%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=13.56%, '
'sys=47.93%, ctx=544144, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=544287,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=38.0MiB/s (39.8MB/s), '
'38.0MiB/s-38.0MiB/s '
'(39.8MB/s-39.8MB/s), io=266MiB '
'(279MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.output'),
'state': {'begin': 1666744108.9835074,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7877578735351562,
'end': 1666744109.7712653,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.state')},
'cmd_58': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.output'),
'state': {'begin': 1666744110.759508,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7883443832397461,
'end': 1666744111.5478523,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.state')},
'cmd_59': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.output'),
'state': {'begin': 1666744111.5498905,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.059826374053955,
'end': 1666744122.609717,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.state')},
'cmd_60': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751211,\n'
' "timestamp_ms" : '
'1666751211808,\n'
' "time" : "Wed Oct 26 04:26:51 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 277933056,\n'
' "io_kbytes" : 271419,\n'
' "bw_bytes" : 39699050,\n'
' "bw" : 38768,\n'
' "iops" : 77537.208970,\n'
' "runtime" : 7001,\n'
' "total_ios" : 542838,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2625,\n'
' "max" : 47760,\n'
' "mean" : 2836.992208,\n'
' "stddev" : 284.296938,\n'
' "N" : 542838\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 311,\n'
' "max" : 110006,\n'
' "mean" : 9445.447319,\n'
' "stddev" : '
'1220.518444,\n'
' "N" : 542838,\n'
' "percentile" : {\n'
' "1.000000" : 9024,\n'
' "5.000000" : 9152,\n'
' "10.000000" : 9152,\n'
' "20.000000" : 9152,\n'
' "30.000000" : 9280,\n'
' "40.000000" : 9280,\n'
' "50.000000" : 9280,\n'
' "60.000000" : 9280,\n'
' "70.000000" : 9408,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9408,\n'
' "95.000000" : 9664,\n'
' "99.000000" : 15168,\n'
' "99.500000" : 15808,\n'
' "99.900000" : 20096,\n'
' "99.950000" : 27264,\n'
' "99.990000" : 56064\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9437,\n'
' "max" : 112812,\n'
' "mean" : 12282.439527,\n'
' "stddev" : '
'1251.244279,\n'
' "N" : 542838\n'
' },\n'
' "bw_min" : 38539,\n'
' "bw_max" : 38959,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'38780.285714,\n'
' "bw_dev" : 105.847807,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 77078,\n'
' "iops_max" : 77918,\n'
' "iops_mean" : '
'77560.642857,\n'
' "iops_stddev" : '
'211.751526,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.314286,\n'
' "sys_cpu" : 48.014286,\n'
' "ctx" : 542695,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010685,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.858547,\n'
' "20" : 3.030923,\n'
' "50" : 0.083819,\n'
' "100" : 0.015658,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=139510: Wed '
'Oct 26 04:26:51 2022\n'
' read: IOPS=77.5k, BW=37.9MiB/s '
'(39.7MB/s)(265MiB/7001msec)\n'
' slat (nsec): min=2625, '
'max=47760, avg=2836.99, '
'stdev=284.30\n'
' clat (nsec): min=311, '
'max=110006, avg=9445.45, '
'stdev=1220.52\n'
' lat (usec): min=9, max=112, '
'avg=12.28, stdev= 1.25\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9024], 5.00th=[ '
'9152], 10.00th=[ 9152], 20.00th=[ '
'9152],\n'
' | 30.00th=[ 9280], 40.00th=[ '
'9280], 50.00th=[ 9280], 60.00th=[ '
'9280],\n'
' | 70.00th=[ 9408], 80.00th=[ '
'9408], 90.00th=[ 9408], 95.00th=[ '
'9664],\n'
' | 99.00th=[15168], '
'99.50th=[15808], 99.90th=[20096], '
'99.95th=[27264],\n'
' | 99.99th=[56064]\n'
' bw ( KiB/s): min=38539, '
'max=38959, per=100.00%, '
'avg=38780.29, stdev=105.85, '
'samples=14\n'
' iops : min=77078, '
'max=77918, avg=77560.64, '
'stdev=211.75, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 10=96.86%, '
'20=3.03%, 50=0.08%, 100=0.02%, '
'250=0.01%\n'
' cpu : usr=13.31%, '
'sys=48.01%, ctx=542695, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=542838,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=37.9MiB/s (39.7MB/s), '
'37.9MiB/s-37.9MiB/s '
'(39.7MB/s-39.7MB/s), io=265MiB '
'(278MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.output'),
'state': {'begin': 1666744122.6130521,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7879660129547119,
'end': 1666744123.4010181,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.state')},
'cmd_61': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.output'),
'state': {'begin': 1666744124.3926356,
'cmd': 'rm '
'/tmp/fio-output_512_1_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7812538146972656,
'end': 1666744125.1738894,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.state')},
'cmd_62': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.output'),
'state': {'begin': 1666744125.1761858,
'cmd': '/opt/fio/bin/fio '
'--name="io_uring-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="io_uring" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_io_uring-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.060308694839478,
'end': 1666744136.2364945,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.state')},
'cmd_63': {'output': 'io_uring-xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=xnvme, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751225,\n'
' "timestamp_ms" : '
'1666751225436,\n'
' "time" : "Wed Oct 26 04:27:05 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"io_uring-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"io_uring-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"io_uring",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 275073536,\n'
' "io_kbytes" : 268626,\n'
' "bw_bytes" : 39290606,\n'
' "bw" : 38369,\n'
' "iops" : 76739.465791,\n'
' "runtime" : 7001,\n'
' "total_ios" : 537253,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2655,\n'
' "max" : 33883,\n'
' "mean" : 2914.394468,\n'
' "stddev" : 191.365400,\n'
' "N" : 537253\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 301,\n'
' "max" : 98825,\n'
' "mean" : 9502.469501,\n'
' "stddev" : '
'1206.508405,\n'
' "N" : 537253,\n'
' "percentile" : {\n'
' "1.000000" : 9024,\n'
' "5.000000" : 9152,\n'
' "10.000000" : 9152,\n'
' "20.000000" : 9280,\n'
' "30.000000" : 9280,\n'
' "40.000000" : 9280,\n'
' "50.000000" : 9408,\n'
' "60.000000" : 9408,\n'
' "70.000000" : 9408,\n'
' "80.000000" : 9408,\n'
' "90.000000" : 9536,\n'
' "95.000000" : 9664,\n'
' "99.000000" : 15168,\n'
' "99.500000" : 15680,\n'
' "99.900000" : 19840,\n'
' "99.950000" : 25472,\n'
' "99.990000" : 57600\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9358,\n'
' "max" : 101720,\n'
' "mean" : 12416.863969,\n'
' "stddev" : '
'1220.253756,\n'
' "N" : 537253\n'
' },\n'
' "bw_min" : 38246,\n'
' "bw_max" : 38615,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'38379.428571,\n'
' "bw_dev" : 100.427329,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 76492,\n'
' "iops_max" : 77230,\n'
' "iops_mean" : '
'76758.857143,\n'
' "iops_stddev" : '
'200.854657,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.200000,\n'
' "sys_cpu" : 46.171429,\n'
' "ctx" : 537127,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.011726,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 96.830916,\n'
' "20" : 3.088675,\n'
' "50" : 0.051372,\n'
' "100" : 0.017310,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'io_uring-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=139549: Wed '
'Oct 26 04:27:05 2022\n'
' read: IOPS=76.7k, BW=37.5MiB/s '
'(39.3MB/s)(262MiB/7001msec)\n'
' slat (nsec): min=2655, '
'max=33883, avg=2914.39, '
'stdev=191.37\n'
' clat (nsec): min=301, '
'max=98825, avg=9502.47, '
'stdev=1206.51\n'
' lat (usec): min=9, max=101, '
'avg=12.42, stdev= 1.22\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 9024], 5.00th=[ '
'9152], 10.00th=[ 9152], 20.00th=[ '
'9280],\n'
' | 30.00th=[ 9280], 40.00th=[ '
'9280], 50.00th=[ 9408], 60.00th=[ '
'9408],\n'
' | 70.00th=[ 9408], 80.00th=[ '
'9408], 90.00th=[ 9536], 95.00th=[ '
'9664],\n'
' | 99.00th=[15168], '
'99.50th=[15680], 99.90th=[19840], '
'99.95th=[25472],\n'
' | 99.99th=[57600]\n'
' bw ( KiB/s): min=38246, '
'max=38615, per=100.00%, '
'avg=38379.43, stdev=100.43, '
'samples=14\n'
' iops : min=76492, '
'max=77230, avg=76758.86, '
'stdev=200.85, samples=14\n'
' lat (nsec) : 500=0.01%\n'
' lat (usec) : 10=96.83%, '
'20=3.09%, 50=0.05%, 100=0.02%\n'
' cpu : usr=15.20%, '
'sys=46.17%, ctx=537127, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=537253,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=37.5MiB/s (39.3MB/s), '
'37.5MiB/s-37.5MiB/s '
'(39.3MB/s-39.3MB/s), io=262MiB '
'(275MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.output'),
'state': {'begin': 1666744136.2393565,
'cmd': 'cat '
'/tmp/fio-output_512_1_io_uring-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7884199619293213,
'end': 1666744137.0277765,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.state')},
'cmd_64': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.output'),
'state': {'begin': 1666744138.0143127,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7859430313110352,
'end': 1666744138.8002558,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.state')},
'cmd_65': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.output'),
'state': {'begin': 1666744138.8029006,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-reference_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.079595804214478,
'end': 1666744149.8824964,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.state')},
'cmd_66': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751239,\n'
' "timestamp_ms" : '
'1666751239082,\n'
' "time" : "Wed Oct 26 04:27:19 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 285667328,\n'
' "io_kbytes" : 278972,\n'
' "bw_bytes" : 40803789,\n'
' "bw" : 39847,\n'
' "iops" : 79694.900728,\n'
' "runtime" : 7001,\n'
' "total_ios" : 557944,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2726,\n'
' "max" : 35376,\n'
' "mean" : 2949.894054,\n'
' "stddev" : 248.175186,\n'
' "N" : 557944\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 801,\n'
' "max" : 72345,\n'
' "mean" : 8997.065191,\n'
' "stddev" : '
'1285.566522,\n'
' "N" : 557944,\n'
' "percentile" : {\n'
' "1.000000" : 8640,\n'
' "5.000000" : 8640,\n'
' "10.000000" : 8640,\n'
' "20.000000" : 8768,\n'
' "30.000000" : 8768,\n'
' "40.000000" : 8768,\n'
' "50.000000" : 8768,\n'
' "60.000000" : 8896,\n'
' "70.000000" : 8896,\n'
' "80.000000" : 8896,\n'
' "90.000000" : 9024,\n'
' "95.000000" : 9152,\n'
' "99.000000" : 15168,\n'
' "99.500000" : 15808,\n'
' "99.900000" : 19328,\n'
' "99.950000" : 26240,\n'
' "99.990000" : 55552\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9508,\n'
' "max" : 75291,\n'
' "mean" : 11946.959245,\n'
' "stddev" : '
'1309.323831,\n'
' "N" : 557944\n'
' },\n'
' "bw_min" : 39689,\n'
' "bw_max" : 39936,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'39863.642857,\n'
' "bw_dev" : 67.919191,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 79378,\n'
' "iops_max" : 79872,\n'
' "iops_mean" : '
'79727.357143,\n'
' "iops_stddev" : '
'135.822060,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 14.842857,\n'
' "sys_cpu" : 45.585714,\n'
' "ctx" : 557800,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 97.375543,\n'
' "20" : 2.532512,\n'
' "50" : 0.062013,\n'
' "100" : 0.019536,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 799685,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5026,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5026,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=139589: Wed '
'Oct 26 04:27:19 2022\n'
' read: IOPS=79.7k, BW=38.9MiB/s '
'(40.8MB/s)(272MiB/7001msec)\n'
' slat (nsec): min=2726, '
'max=35376, avg=2949.89, '
'stdev=248.18\n'
' clat (nsec): min=801, '
'max=72345, avg=8997.07, '
'stdev=1285.57\n'
' lat (nsec): min=9508, '
'max=75291, avg=11946.96, '
'stdev=1309.32\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8640], 5.00th=[ '
'8640], 10.00th=[ 8640], 20.00th=[ '
'8768],\n'
' | 30.00th=[ 8768], 40.00th=[ '
'8768], 50.00th=[ 8768], 60.00th=[ '
'8896],\n'
' | 70.00th=[ 8896], 80.00th=[ '
'8896], 90.00th=[ 9024], 95.00th=[ '
'9152],\n'
' | 99.00th=[15168], '
'99.50th=[15808], 99.90th=[19328], '
'99.95th=[26240],\n'
' | 99.99th=[55552]\n'
' bw ( KiB/s): min=39689, '
'max=39936, per=100.00%, '
'avg=39863.64, stdev=67.92, '
'samples=14\n'
' iops : min=79378, '
'max=79872, avg=79727.36, '
'stdev=135.82, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'10=97.38%, 20=2.53%, 50=0.06%, '
'100=0.02%\n'
' cpu : usr=14.84%, '
'sys=45.59%, ctx=557800, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=557944,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=38.9MiB/s (40.8MB/s), '
'38.9MiB/s-38.9MiB/s '
'(40.8MB/s-40.8MB/s), io=272MiB '
'(286MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=799685/0, '
'merge=0/0, ticks=5026/0, '
'in_queue=5026, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.output'),
'state': {'begin': 1666744149.8855157,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-reference_0.txt',
'cwd': 'None',
'elapsed': 0.7835009098052979,
'end': 1666744150.6690166,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.state')},
'cmd_67': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.output'),
'state': {'begin': 1666744151.6574724,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7808341979980469,
'end': 1666744152.4383066,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.state')},
'cmd_68': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.output'),
'state': {'begin': 1666744152.440717,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-reference_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.063346147537231,
'end': 1666744163.5040631,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.state')},
'cmd_69': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751252,\n'
' "timestamp_ms" : '
'1666751252703,\n'
' "time" : "Wed Oct 26 04:27:32 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 435968000,\n'
' "io_kbytes" : 425750,\n'
' "bw_bytes" : 62272246,\n'
' "bw" : 60812,\n'
' "iops" : 121625.482074,\n'
' "runtime" : 7001,\n'
' "total_ios" : 851500,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1543,\n'
' "max" : 14407,\n'
' "mean" : 1658.127843,\n'
' "stddev" : 52.610402,\n'
' "N" : 851500\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 410,\n'
' "max" : 96491,\n'
' "mean" : 6259.916447,\n'
' "stddev" : '
'1052.788853,\n'
' "N" : 851500,\n'
' "percentile" : {\n'
' "1.000000" : 5920,\n'
' "5.000000" : 5984,\n'
' "10.000000" : 5984,\n'
' "20.000000" : 6048,\n'
' "30.000000" : 6048,\n'
' "40.000000" : 6112,\n'
' "50.000000" : 6112,\n'
' "60.000000" : 6176,\n'
' "70.000000" : 6176,\n'
' "80.000000" : 6240,\n'
' "90.000000" : 6368,\n'
' "95.000000" : 6432,\n'
' "99.000000" : 11968,\n'
' "99.500000" : 12480,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 56064\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 6091,\n'
' "max" : 98144,\n'
' "mean" : 7918.044290,\n'
' "stddev" : '
'1054.920677,\n'
' "N" : 851500\n'
' },\n'
' "bw_min" : 60701,\n'
' "bw_max" : 61033,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'60838.642857,\n'
' "bw_dev" : 110.521566,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 121402,\n'
' "iops_max" : 122066,\n'
' "iops_mean" : '
'121677.428571,\n'
' "iops_stddev" : '
'221.098208,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 13.700000,\n'
' "sys_cpu" : 36.571429,\n'
' "ctx" : 851445,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.010000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 98.463065,\n'
' "20" : 1.515326,\n'
' "50" : 0.010000,\n'
' "100" : 0.011861,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 1221975,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5983,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5983,\n'
' "util" : 99.088748\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=139627: Wed '
'Oct 26 04:27:32 2022\n'
' read: IOPS=122k, BW=59.4MiB/s '
'(62.3MB/s)(416MiB/7001msec)\n'
' slat (nsec): min=1543, '
'max=14407, avg=1658.13, '
'stdev=52.61\n'
' clat (nsec): min=410, '
'max=96491, avg=6259.92, '
'stdev=1052.79\n'
' lat (nsec): min=6091, '
'max=98144, avg=7918.04, '
'stdev=1054.92\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 5920], 5.00th=[ '
'5984], 10.00th=[ 5984], 20.00th=[ '
'6048],\n'
' | 30.00th=[ 6048], 40.00th=[ '
'6112], 50.00th=[ 6112], 60.00th=[ '
'6176],\n'
' | 70.00th=[ 6176], 80.00th=[ '
'6240], 90.00th=[ 6368], 95.00th=[ '
'6432],\n'
' | 99.00th=[11968], '
'99.50th=[12480], 99.90th=[15552], '
'99.95th=[15680],\n'
' | 99.99th=[56064]\n'
' bw ( KiB/s): min=60701, '
'max=61033, per=100.00%, '
'avg=60838.64, stdev=110.52, '
'samples=14\n'
' iops : min=121402, '
'max=122066, avg=121677.43, '
'stdev=221.10, samples=14\n'
' lat (nsec) : 500=0.01%, '
'750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=98.46%, 20=1.52%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=13.70%, '
'sys=36.57%, ctx=851445, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=851500,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=59.4MiB/s (62.3MB/s), '
'59.4MiB/s-59.4MiB/s '
'(62.3MB/s-62.3MB/s), io=416MiB '
'(436MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=1221975/0, '
'merge=0/0, ticks=5983/0, '
'in_queue=5983, util=99.09%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.output'),
'state': {'begin': 1666744163.5061615,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-reference_1.txt',
'cwd': 'None',
'elapsed': 0.7857673168182373,
'end': 1666744164.2919288,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.state')},
'cmd_70': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.output'),
'state': {'begin': 1666744165.280862,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.7872707843780518,
'end': 1666744166.0681329,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.state')},
'cmd_71': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.output'),
'state': {'begin': 1666744166.0701516,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-reference" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="libaio" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-reference_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.097615957260132,
'end': 1666744177.1677675,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.state')},
'cmd_72': {'output': 'libaio-reference: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=libaio, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751266,\n'
' "timestamp_ms" : '
'1666751266367,\n'
' "time" : "Wed Oct 26 04:27:46 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-reference",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-reference",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : "libaio",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 281868800,\n'
' "io_kbytes" : 275262,\n'
' "bw_bytes" : 40261219,\n'
' "bw" : 39317,\n'
' "iops" : 78635.194972,\n'
' "runtime" : 7001,\n'
' "total_ios" : 550525,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2765,\n'
' "max" : 34235,\n'
' "mean" : 2952.005994,\n'
' "stddev" : 196.975536,\n'
' "N" : 550525\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 791,\n'
' "max" : 76494,\n'
' "mean" : 9169.578695,\n'
' "stddev" : '
'1235.513611,\n'
' "N" : 550525,\n'
' "percentile" : {\n'
' "1.000000" : 8768,\n'
' "5.000000" : 8768,\n'
' "10.000000" : 8896,\n'
' "20.000000" : 8896,\n'
' "30.000000" : 8896,\n'
' "40.000000" : 9024,\n'
' "50.000000" : 9024,\n'
' "60.000000" : 9024,\n'
' "70.000000" : 9024,\n'
' "80.000000" : 9152,\n'
' "90.000000" : 9152,\n'
' "95.000000" : 9280,\n'
' "99.000000" : 15040,\n'
' "99.500000" : 15680,\n'
' "99.900000" : 19328,\n'
' "99.950000" : 23936,\n'
' "99.990000" : 57600\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9348,\n'
' "max" : 79429,\n'
' "mean" : 12121.584689,\n'
' "stddev" : '
'1250.717136,\n'
' "N" : 550525\n'
' },\n'
' "bw_min" : 39172,\n'
' "bw_max" : 39524,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'39335.142857,\n'
' "bw_dev" : 102.275538,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 78344,\n'
' "iops_max" : 79048,\n'
' "iops_mean" : '
'78670.428571,\n'
' "iops_stddev" : '
'204.625630,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.157143,\n'
' "sys_cpu" : 44.100000,\n'
' "ctx" : 550395,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 97.186867,\n'
' "20" : 2.744653,\n'
' "50" : 0.040507,\n'
' "100" : 0.018709,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ],\n'
' "disk_util" : [\n'
' {\n'
' "name" : "nvme0n1",\n'
' "read_ios" : 790291,\n'
' "write_ios" : 0,\n'
' "read_merges" : 0,\n'
' "write_merges" : 0,\n'
' "read_ticks" : 5056,\n'
' "write_ticks" : 0,\n'
' "in_queue" : 5056,\n'
' "util" : 99.058940\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-reference: (groupid=0, '
'jobs=1): err= 0: pid=139665: Wed '
'Oct 26 04:27:46 2022\n'
' read: IOPS=78.6k, BW=38.4MiB/s '
'(40.3MB/s)(269MiB/7001msec)\n'
' slat (nsec): min=2765, '
'max=34235, avg=2952.01, '
'stdev=196.98\n'
' clat (nsec): min=791, '
'max=76494, avg=9169.58, '
'stdev=1235.51\n'
' lat (nsec): min=9348, '
'max=79429, avg=12121.58, '
'stdev=1250.72\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8768], 5.00th=[ '
'8768], 10.00th=[ 8896], 20.00th=[ '
'8896],\n'
' | 30.00th=[ 8896], 40.00th=[ '
'9024], 50.00th=[ 9024], 60.00th=[ '
'9024],\n'
' | 70.00th=[ 9024], 80.00th=[ '
'9152], 90.00th=[ 9152], 95.00th=[ '
'9280],\n'
' | 99.00th=[15040], '
'99.50th=[15680], 99.90th=[19328], '
'99.95th=[23936],\n'
' | 99.99th=[57600]\n'
' bw ( KiB/s): min=39172, '
'max=39524, per=100.00%, '
'avg=39335.14, stdev=102.28, '
'samples=14\n'
' iops : min=78344, '
'max=79048, avg=78670.43, '
'stdev=204.63, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'10=97.19%, 20=2.74%, 50=0.04%, '
'100=0.02%\n'
' cpu : usr=16.16%, '
'sys=44.10%, ctx=550395, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=550525,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=38.4MiB/s (40.3MB/s), '
'38.4MiB/s-38.4MiB/s '
'(40.3MB/s-40.3MB/s), io=269MiB '
'(282MB), run=7001-7001msec\n'
'\n'
'Disk stats (read/write):\n'
' nvme0n1: ios=790291/0, '
'merge=0/0, ticks=5056/0, '
'in_queue=5056, util=99.06%\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.output'),
'state': {'begin': 1666744177.169167,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-reference_2.txt',
'cwd': 'None',
'elapsed': 0.781212568283081,
'end': 1666744177.9503796,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.state')},
'cmd_73': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.output'),
'state': {'begin': 1666744179.963178,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7831017971038818,
'end': 1666744180.7462797,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.state')},
'cmd_74': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.output'),
'state': {'begin': 1666744180.7478044,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_aio_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.523125410079956,
'end': 1666744192.2709298,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.state')},
'cmd_75': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751281,\n'
' "timestamp_ms" : '
'1666751281368,\n'
' "time" : "Wed Oct 26 04:28:01 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 568576000,\n'
' "io_kbytes" : 555250,\n'
' "bw_bytes" : 81225142,\n'
' "bw" : 79321,\n'
' "iops" : 158642.857143,\n'
' "runtime" : 7000,\n'
' "total_ios" : 1110500,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1613,\n'
' "max" : 37350,\n'
' "mean" : 1678.867968,\n'
' "stddev" : 146.532978,\n'
' "N" : 1110500\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 190,\n'
' "max" : 115837,\n'
' "mean" : 4397.280773,\n'
' "stddev" : '
'1142.935701,\n'
' "N" : 1110500,\n'
' "percentile" : {\n'
' "1.000000" : 4128,\n'
' "5.000000" : 4128,\n'
' "10.000000" : 4128,\n'
' "20.000000" : 4192,\n'
' "30.000000" : 4192,\n'
' "40.000000" : 4192,\n'
' "50.000000" : 4256,\n'
' "60.000000" : 4256,\n'
' "70.000000" : 4320,\n'
' "80.000000" : 4320,\n'
' "90.000000" : 4384,\n'
' "95.000000" : 4576,\n'
' "99.000000" : 9792,\n'
' "99.500000" : 11328,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5730,\n'
' "max" : 117510,\n'
' "mean" : 6076.148740,\n'
' "stddev" : '
'1154.189344,\n'
' "N" : 1110500\n'
' },\n'
' "bw_min" : 79120,\n'
' "bw_max" : 79623,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'79328.214286,\n'
' "bw_dev" : 158.211241,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 158240,\n'
' "iops_max" : 159246,\n'
' "iops_mean" : '
'158656.500000,\n'
' "iops_stddev" : '
'316.477913,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 76.614286,\n'
' "sys_cpu" : 23.357143,\n'
' "ctx" : 26,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 99.038451,\n'
' "20" : 0.941108,\n'
' "50" : 0.010000,\n'
' "100" : 0.012517,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=139713: Wed '
'Oct 26 04:28:01 2022\n'
' read: IOPS=159k, BW=77.5MiB/s '
'(81.2MB/s)(542MiB/7000msec)\n'
' slat (nsec): min=1613, '
'max=37350, avg=1678.87, '
'stdev=146.53\n'
' clat (nsec): min=190, '
'max=115837, avg=4397.28, '
'stdev=1142.94\n'
' lat (usec): min=5, max=117, '
'avg= 6.08, stdev= 1.15\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4128], 5.00th=[ '
'4128], 10.00th=[ 4128], 20.00th=[ '
'4192],\n'
' | 30.00th=[ 4192], 40.00th=[ '
'4192], 50.00th=[ 4256], 60.00th=[ '
'4256],\n'
' | 70.00th=[ 4320], 80.00th=[ '
'4320], 90.00th=[ 4384], 95.00th=[ '
'4576],\n'
' | 99.00th=[ 9792], '
'99.50th=[11328], 99.90th=[15552], '
'99.95th=[15680],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=79120, '
'max=79623, per=100.00%, '
'avg=79328.21, stdev=158.21, '
'samples=14\n'
' iops : min=158240, '
'max=159246, avg=158656.50, '
'stdev=316.48, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=99.04%, 20=0.94%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%, '
'250=0.01%\n'
' cpu : usr=76.61%, '
'sys=23.36%, ctx=26, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1110500,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=77.5MiB/s (81.2MB/s), '
'77.5MiB/s-77.5MiB/s '
'(81.2MB/s-81.2MB/s), io=542MiB '
'(569MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.output'),
'state': {'begin': 1666744192.2780962,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_aio_0.txt',
'cwd': 'None',
'elapsed': 0.7858402729034424,
'end': 1666744193.0639365,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.state')},
'cmd_76': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.output'),
'state': {'begin': 1666744195.081775,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.7862448692321777,
'end': 1666744195.8680198,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.state')},
'cmd_77': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.output'),
'state': {'begin': 1666744195.8700564,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_aio_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.520851612091064,
'end': 1666744207.390908,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.state')},
'cmd_78': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751296,\n'
' "timestamp_ms" : '
'1666751296479,\n'
' "time" : "Wed Oct 26 04:28:16 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 569852416,\n'
' "io_kbytes" : 556496,\n'
' "bw_bytes" : 81407488,\n'
' "bw" : 79499,\n'
' "iops" : 158999.000000,\n'
' "runtime" : 7000,\n'
' "total_ios" : 1112993,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1623,\n'
' "max" : 25918,\n'
' "mean" : 1684.411642,\n'
' "stddev" : 128.915050,\n'
' "N" : 1112993\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 210,\n'
' "max" : 76463,\n'
' "mean" : 4380.684294,\n'
' "stddev" : '
'1145.616557,\n'
' "N" : 1112993,\n'
' "percentile" : {\n'
' "1.000000" : 4128,\n'
' "5.000000" : 4128,\n'
' "10.000000" : 4128,\n'
' "20.000000" : 4128,\n'
' "30.000000" : 4192,\n'
' "40.000000" : 4192,\n'
' "50.000000" : 4256,\n'
' "60.000000" : 4256,\n'
' "70.000000" : 4256,\n'
' "80.000000" : 4320,\n'
' "90.000000" : 4384,\n'
' "95.000000" : 4512,\n'
' "99.000000" : 9792,\n'
' "99.500000" : 11200,\n'
' "99.900000" : 15424,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 57600\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5731,\n'
' "max" : 78166,\n'
' "mean" : 6065.095936,\n'
' "stddev" : '
'1154.036926,\n'
' "N" : 1112993\n'
' },\n'
' "bw_min" : 79322,\n'
' "bw_max" : 79715,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'79509.214286,\n'
' "bw_dev" : 119.767196,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 158644,\n'
' "iops_max" : 159430,\n'
' "iops_mean" : '
'159018.571429,\n'
' "iops_stddev" : '
'239.596364,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 76.442857,\n'
' "sys_cpu" : 23.528571,\n'
' "ctx" : 26,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 99.035214,\n'
' "20" : 0.943762,\n'
' "50" : 0.010000,\n'
' "100" : 0.013208,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=139762: Wed '
'Oct 26 04:28:16 2022\n'
' read: IOPS=159k, BW=77.6MiB/s '
'(81.4MB/s)(543MiB/7000msec)\n'
' slat (nsec): min=1623, '
'max=25918, avg=1684.41, '
'stdev=128.92\n'
' clat (nsec): min=210, '
'max=76463, avg=4380.68, '
'stdev=1145.62\n'
' lat (nsec): min=5731, '
'max=78166, avg=6065.10, '
'stdev=1154.04\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4128], 5.00th=[ '
'4128], 10.00th=[ 4128], 20.00th=[ '
'4128],\n'
' | 30.00th=[ 4192], 40.00th=[ '
'4192], 50.00th=[ 4256], 60.00th=[ '
'4256],\n'
' | 70.00th=[ 4256], 80.00th=[ '
'4320], 90.00th=[ 4384], 95.00th=[ '
'4512],\n'
' | 99.00th=[ 9792], '
'99.50th=[11200], 99.90th=[15424], '
'99.95th=[15680],\n'
' | 99.99th=[57600]\n'
' bw ( KiB/s): min=79322, '
'max=79715, per=100.00%, '
'avg=79509.21, stdev=119.77, '
'samples=14\n'
' iops : min=158644, '
'max=159430, avg=159018.57, '
'stdev=239.60, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=99.04%, 20=0.94%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=76.44%, '
'sys=23.53%, ctx=26, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1112993,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=77.6MiB/s (81.4MB/s), '
'77.6MiB/s-77.6MiB/s '
'(81.4MB/s-81.4MB/s), io=543MiB '
'(570MB), run=7000-7000msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.output'),
'state': {'begin': 1666744207.3945053,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_aio_1.txt',
'cwd': 'None',
'elapsed': 0.8608551025390625,
'end': 1666744208.2553604,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.state')},
'cmd_79': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.output'),
'state': {'begin': 1666744210.2829776,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7859225273132324,
'end': 1666744211.0689,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.state')},
'cmd_80': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.output'),
'state': {'begin': 1666744211.0708146,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_aio" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_aio_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.525362253189087,
'end': 1666744222.5961769,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.state')},
'cmd_81': {'output': 'libaio-bdev_aio: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751311,\n'
' "timestamp_ms" : '
'1666751311692,\n'
' "time" : "Wed Oct 26 04:28:31 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_aio",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_aio",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_aio_foo.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 570132480,\n'
' "io_kbytes" : 556770,\n'
' "bw_bytes" : 81435863,\n'
' "bw" : 79527,\n'
' "iops" : 159054.277960,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1113539,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1603,\n'
' "max" : 60874,\n'
' "mean" : 1654.208657,\n'
' "stddev" : 140.431430,\n'
' "N" : 1113540\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 200,\n'
' "max" : 114164,\n'
' "mean" : 4398.236214,\n'
' "stddev" : '
'1110.230874,\n'
' "N" : 1113539,\n'
' "percentile" : {\n'
' "1.000000" : 4128,\n'
' "5.000000" : 4128,\n'
' "10.000000" : 4192,\n'
' "20.000000" : 4192,\n'
' "30.000000" : 4192,\n'
' "40.000000" : 4192,\n'
' "50.000000" : 4256,\n'
' "60.000000" : 4256,\n'
' "70.000000" : 4320,\n'
' "80.000000" : 4320,\n'
' "90.000000" : 4384,\n'
' "95.000000" : 4512,\n'
' "99.000000" : 9920,\n'
' "99.500000" : 10560,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5730,\n'
' "max" : 115827,\n'
' "mean" : 6052.444891,\n'
' "stddev" : '
'1120.267316,\n'
' "N" : 1113539\n'
' },\n'
' "bw_min" : 79426,\n'
' "bw_max" : 79846,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'79546.428571,\n'
' "bw_dev" : 125.765175,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 158852,\n'
' "iops_max" : 159693,\n'
' "iops_mean" : '
'159092.785714,\n'
' "iops_stddev" : '
'251.830889,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 76.100000,\n'
' "sys_cpu" : 23.871429,\n'
' "ctx" : 25,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.010000,\n'
' "500" : 0.010000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 99.023653,\n'
' "20" : 0.955692,\n'
' "50" : 0.010000,\n'
' "100" : 0.011674,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_aio: (groupid=0, '
'jobs=1): err= 0: pid=139811: Wed '
'Oct 26 04:28:31 2022\n'
' read: IOPS=159k, BW=77.7MiB/s '
'(81.4MB/s)(544MiB/7001msec)\n'
' slat (nsec): min=1603, '
'max=60874, avg=1654.21, '
'stdev=140.43\n'
' clat (nsec): min=200, '
'max=114164, avg=4398.24, '
'stdev=1110.23\n'
' lat (usec): min=5, max=115, '
'avg= 6.05, stdev= 1.12\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4128], 5.00th=[ '
'4128], 10.00th=[ 4192], 20.00th=[ '
'4192],\n'
' | 30.00th=[ 4192], 40.00th=[ '
'4192], 50.00th=[ 4256], 60.00th=[ '
'4256],\n'
' | 70.00th=[ 4320], 80.00th=[ '
'4320], 90.00th=[ 4384], 95.00th=[ '
'4512],\n'
' | 99.00th=[ 9920], '
'99.50th=[10560], 99.90th=[15552], '
'99.95th=[15680],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=79426, '
'max=79846, per=100.00%, '
'avg=79546.43, stdev=125.77, '
'samples=14\n'
' iops : min=158852, '
'max=159693, avg=159092.79, '
'stdev=251.83, samples=14\n'
' lat (nsec) : 250=0.01%, '
'500=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=99.02%, 20=0.96%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%, '
'250=0.01%\n'
' cpu : usr=76.10%, '
'sys=23.87%, ctx=25, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1113539,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=77.7MiB/s (81.4MB/s), '
'77.7MiB/s-77.7MiB/s '
'(81.4MB/s-81.4MB/s), io=544MiB '
'(570MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.output'),
'state': {'begin': 1666744222.599019,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_aio_2.txt',
'cwd': 'None',
'elapsed': 0.7893621921539307,
'end': 1666744223.3883812,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.state')},
'cmd_82': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.output'),
'state': {'begin': 1666744225.4102664,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7753422260284424,
'end': 1666744226.1856086,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.state')},
'cmd_83': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.output'),
'state': {'begin': 1666744226.187541,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.548415660858154,
'end': 1666744237.7359567,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.state')},
'cmd_84': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751326,\n'
' "timestamp_ms" : '
'1666751326817,\n'
' "time" : "Wed Oct 26 04:28:46 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 549307392,\n'
' "io_kbytes" : 536433,\n'
' "bw_bytes" : 78461275,\n'
' "bw" : 76622,\n'
' "iops" : 153244.679332,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1072866,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1623,\n'
' "max" : 14217,\n'
' "mean" : 1678.038774,\n'
' "stddev" : 119.906182,\n'
' "N" : 1072866\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 591,\n'
' "max" : 118713,\n'
' "mean" : 4583.107354,\n'
' "stddev" : '
'1160.723079,\n'
' "N" : 1072866,\n'
' "percentile" : {\n'
' "1.000000" : 4320,\n'
' "5.000000" : 4320,\n'
' "10.000000" : 4320,\n'
' "20.000000" : 4320,\n'
' "30.000000" : 4320,\n'
' "40.000000" : 4320,\n'
' "50.000000" : 4384,\n'
' "60.000000" : 4384,\n'
' "70.000000" : 4640,\n'
' "80.000000" : 4640,\n'
' "90.000000" : 4704,\n'
' "95.000000" : 4768,\n'
' "99.000000" : 9792,\n'
' "99.500000" : 10816,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 59136\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5881,\n'
' "max" : 120406,\n'
' "mean" : 6261.146127,\n'
' "stddev" : '
'1168.029557,\n'
' "N" : 1072866\n'
' },\n'
' "bw_min" : 76498,\n'
' "bw_max" : 76736,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'76644.785714,\n'
' "bw_dev" : 66.817983,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 152997,\n'
' "iops_max" : 153472,\n'
' "iops_mean" : '
'153289.642857,\n'
' "iops_stddev" : '
'133.467142,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 36.614286,\n'
' "sys_cpu" : 63.357143,\n'
' "ctx" : 23,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 99.073137,\n'
' "20" : 0.903188,\n'
' "50" : 0.010000,\n'
' "100" : 0.015379,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=139861: Wed '
'Oct 26 04:28:46 2022\n'
' read: IOPS=153k, BW=74.8MiB/s '
'(78.5MB/s)(524MiB/7001msec)\n'
' slat (nsec): min=1623, '
'max=14217, avg=1678.04, '
'stdev=119.91\n'
' clat (nsec): min=591, '
'max=118713, avg=4583.11, '
'stdev=1160.72\n'
' lat (usec): min=5, max=120, '
'avg= 6.26, stdev= 1.17\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4320], 5.00th=[ '
'4320], 10.00th=[ 4320], 20.00th=[ '
'4320],\n'
' | 30.00th=[ 4320], 40.00th=[ '
'4320], 50.00th=[ 4384], 60.00th=[ '
'4384],\n'
' | 70.00th=[ 4640], 80.00th=[ '
'4640], 90.00th=[ 4704], 95.00th=[ '
'4768],\n'
' | 99.00th=[ 9792], '
'99.50th=[10816], 99.90th=[15552], '
'99.95th=[15680],\n'
' | 99.99th=[59136]\n'
' bw ( KiB/s): min=76498, '
'max=76736, per=100.00%, '
'avg=76644.79, stdev=66.82, '
'samples=14\n'
' iops : min=152997, '
'max=153472, avg=153289.64, '
'stdev=133.47, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=99.07%, 20=0.90%, '
'50=0.01%\n'
' lat (usec) : 100=0.02%, '
'250=0.01%\n'
' cpu : usr=36.61%, '
'sys=63.36%, ctx=23, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1072866,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=74.8MiB/s (78.5MB/s), '
'74.8MiB/s-74.8MiB/s '
'(78.5MB/s-78.5MB/s), io=524MiB '
'(549MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.output'),
'state': {'begin': 1666744237.7385774,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7844376564025879,
'end': 1666744238.523015,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.state')},
'cmd_85': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.output'),
'state': {'begin': 1666744240.5375273,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7823998928070068,
'end': 1666744241.3199272,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.state')},
'cmd_86': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.output'),
'state': {'begin': 1666744241.3217847,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.508594512939453,
'end': 1666744252.8303792,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.state')},
'cmd_87': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751341,\n'
' "timestamp_ms" : '
'1666751341933,\n'
' "time" : "Wed Oct 26 04:29:01 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 548251136,\n'
' "io_kbytes" : 535401,\n'
' "bw_bytes" : 78310403,\n'
' "bw" : 76475,\n'
' "iops" : 152950.007142,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1070803,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1633,\n'
' "max" : 14327,\n'
' "mean" : 1697.999735,\n'
' "stddev" : 113.979161,\n'
' "N" : 1070803\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 602,\n'
' "max" : 97693,\n'
' "mean" : 4606.212420,\n'
' "stddev" : '
'1101.921202,\n'
' "N" : 1070803,\n'
' "percentile" : {\n'
' "1.000000" : 4384,\n'
' "5.000000" : 4384,\n'
' "10.000000" : 4384,\n'
' "20.000000" : 4384,\n'
' "30.000000" : 4384,\n'
' "40.000000" : 4448,\n'
' "50.000000" : 4448,\n'
' "60.000000" : 4448,\n'
' "70.000000" : 4448,\n'
' "80.000000" : 4512,\n'
' "90.000000" : 4704,\n'
' "95.000000" : 4768,\n'
' "99.000000" : 9792,\n'
' "99.500000" : 10816,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 57088\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5921,\n'
' "max" : 99376,\n'
' "mean" : 6304.212155,\n'
' "stddev" : '
'1109.513640,\n'
' "N" : 1070803\n'
' },\n'
' "bw_min" : 76350,\n'
' "bw_max" : 76686,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'76496.357143,\n'
' "bw_dev" : 105.615855,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 152700,\n'
' "iops_max" : 153372,\n'
' "iops_mean" : '
'152992.928571,\n'
' "iops_stddev" : '
'211.019673,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 29.985714,\n'
' "sys_cpu" : 69.985714,\n'
' "ctx" : 20,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 99.040907,\n'
' "20" : 0.937054,\n'
' "50" : 0.010000,\n'
' "100" : 0.013168,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=139913: Wed '
'Oct 26 04:29:01 2022\n'
' read: IOPS=153k, BW=74.7MiB/s '
'(78.3MB/s)(523MiB/7001msec)\n'
' slat (nsec): min=1633, '
'max=14327, avg=1698.00, '
'stdev=113.98\n'
' clat (nsec): min=602, '
'max=97693, avg=4606.21, '
'stdev=1101.92\n'
' lat (nsec): min=5921, '
'max=99376, avg=6304.21, '
'stdev=1109.51\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4384], 5.00th=[ '
'4384], 10.00th=[ 4384], 20.00th=[ '
'4384],\n'
' | 30.00th=[ 4384], 40.00th=[ '
'4448], 50.00th=[ 4448], 60.00th=[ '
'4448],\n'
' | 70.00th=[ 4448], 80.00th=[ '
'4512], 90.00th=[ 4704], 95.00th=[ '
'4768],\n'
' | 99.00th=[ 9792], '
'99.50th=[10816], 99.90th=[15552], '
'99.95th=[15680],\n'
' | 99.99th=[57088]\n'
' bw ( KiB/s): min=76350, '
'max=76686, per=100.00%, '
'avg=76496.36, stdev=105.62, '
'samples=14\n'
' iops : min=152700, '
'max=153372, avg=152992.93, '
'stdev=211.02, samples=14\n'
' lat (nsec) : 750=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=99.04%, 20=0.94%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=29.99%, '
'sys=69.99%, ctx=20, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1070803,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=74.7MiB/s (78.3MB/s), '
'74.7MiB/s-74.7MiB/s '
'(78.3MB/s-78.3MB/s), io=523MiB '
'(548MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.output'),
'state': {'begin': 1666744252.833364,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7898769378662109,
'end': 1666744253.623241,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.state')},
'cmd_88': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.output'),
'state': {'begin': 1666744255.6336243,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7826905250549316,
'end': 1666744256.4163148,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.state')},
'cmd_89': {'output': 'TELEMETRY: No legacy callbacks, '
'legacy socket not created\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.output'),
'state': {'begin': 1666744256.4187577,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-bdev_xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="spdk_bdev" '
'--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
'--filename="Nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.516239881515503,
'end': 1666744267.9349976,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.state')},
'cmd_90': {'output': 'libaio-bdev_xnvme: (g=0): '
'rw=randread, bs=(R) 512B-512B, '
'(W) 512B-512B, (T) 512B-512B, '
'ioengine=spdk_bdev, iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751357,\n'
' "timestamp_ms" : '
'1666751357029,\n'
' "time" : "Wed Oct 26 04:29:17 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-bdev_xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : '
'"libaio-bdev_xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"spdk_bdev",\n'
' "spdk_conf" : '
'"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
' "filename" : "Nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 552497664,\n'
' "io_kbytes" : 539548,\n'
' "bw_bytes" : 78916963,\n'
' "bw" : 77067,\n'
' "iops" : 154134.695044,\n'
' "runtime" : 7001,\n'
' "total_ios" : 1079097,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 1613,\n'
' "max" : 25488,\n'
' "mean" : 1684.209908,\n'
' "stddev" : 120.441497,\n'
' "N" : 1079097\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 592,\n'
' "max" : 76343,\n'
' "mean" : 4568.302466,\n'
' "stddev" : '
'1090.399727,\n'
' "N" : 1079097,\n'
' "percentile" : {\n'
' "1.000000" : 4320,\n'
' "5.000000" : 4320,\n'
' "10.000000" : 4320,\n'
' "20.000000" : 4384,\n'
' "30.000000" : 4384,\n'
' "40.000000" : 4384,\n'
' "50.000000" : 4384,\n'
' "60.000000" : 4448,\n'
' "70.000000" : 4448,\n'
' "80.000000" : 4448,\n'
' "90.000000" : 4704,\n'
' "95.000000" : 4768,\n'
' "99.000000" : 9792,\n'
' "99.500000" : 11328,\n'
' "99.900000" : 15552,\n'
' "99.950000" : 15680,\n'
' "99.990000" : 56576\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 5861,\n'
' "max" : 78016,\n'
' "mean" : 6252.512374,\n'
' "stddev" : '
'1098.558092,\n'
' "N" : 1079097\n'
' },\n'
' "bw_min" : 76780,\n'
' "bw_max" : 77307,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'77088.357143,\n'
' "bw_dev" : 198.919236,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 153560,\n'
' "iops_max" : 154614,\n'
' "iops_mean" : '
'154176.714286,\n'
' "iops_stddev" : '
'397.838473,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 29.242857,\n'
' "sys_cpu" : 70.728571,\n'
' "ctx" : 22,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.010000,\n'
' "10" : 99.054580,\n'
' "20" : 0.927349,\n'
' "50" : 0.010000,\n'
' "100" : 0.010842,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-bdev_xnvme: (groupid=0, '
'jobs=1): err= 0: pid=139972: Wed '
'Oct 26 04:29:17 2022\n'
' read: IOPS=154k, BW=75.3MiB/s '
'(78.9MB/s)(527MiB/7001msec)\n'
' slat (nsec): min=1613, '
'max=25488, avg=1684.21, '
'stdev=120.44\n'
' clat (nsec): min=592, '
'max=76343, avg=4568.30, '
'stdev=1090.40\n'
' lat (nsec): min=5861, '
'max=78016, avg=6252.51, '
'stdev=1098.56\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 4320], 5.00th=[ '
'4320], 10.00th=[ 4320], 20.00th=[ '
'4384],\n'
' | 30.00th=[ 4384], 40.00th=[ '
'4384], 50.00th=[ 4384], 60.00th=[ '
'4448],\n'
' | 70.00th=[ 4448], 80.00th=[ '
'4448], 90.00th=[ 4704], 95.00th=[ '
'4768],\n'
' | 99.00th=[ 9792], '
'99.50th=[11328], 99.90th=[15552], '
'99.95th=[15680],\n'
' | 99.99th=[56576]\n'
' bw ( KiB/s): min=76780, '
'max=77307, per=100.00%, '
'avg=77088.36, stdev=198.92, '
'samples=14\n'
' iops : min=153560, '
'max=154614, avg=154176.71, '
'stdev=397.84, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'4=0.01%, 10=99.05%, 20=0.93%, '
'50=0.01%\n'
' lat (usec) : 100=0.01%\n'
' cpu : usr=29.24%, '
'sys=70.73%, ctx=22, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=1079097,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=75.3MiB/s (78.9MB/s), '
'75.3MiB/s-75.3MiB/s '
'(78.9MB/s-78.9MB/s), io=527MiB '
'(552MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.output'),
'state': {'begin': 1666744267.9379344,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7892000675201416,
'end': 1666744268.7271345,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.state')},
'cmd_91': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.output'),
'state': {'begin': 1666744269.753349,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7843270301818848,
'end': 1666744270.537676,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.state')},
'cmd_92': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.output'),
'state': {'begin': 1666744270.5395074,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-xnvme_0.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.08480453491211,
'end': 1666744281.624312,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.state')},
'cmd_93': {'output': 'libaio-xnvme: (g=0): rw=randread, '
'bs=(R) 512B-512B, (W) 512B-512B, '
'(T) 512B-512B, ioengine=xnvme, '
'iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751370,\n'
' "timestamp_ms" : '
'1666751370821,\n'
' "time" : "Wed Oct 26 04:29:30 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : "libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 263074816,\n'
' "io_kbytes" : 256909,\n'
' "bw_bytes" : 37576748,\n'
' "bw" : 36696,\n'
' "iops" : 73392.086845,\n'
' "runtime" : 7001,\n'
' "total_ios" : 513818,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2856,\n'
' "max" : 24596,\n'
' "mean" : 3174.835060,\n'
' "stddev" : 192.695012,\n'
' "N" : 513818\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 942,\n'
' "max" : 70743,\n'
' "mean" : 9889.292977,\n'
' "stddev" : '
'1172.701180,\n'
' "N" : 513818,\n'
' "percentile" : {\n'
' "1.000000" : 8512,\n'
' "5.000000" : 9536,\n'
' "10.000000" : 9664,\n'
' "20.000000" : 9664,\n'
' "30.000000" : 9792,\n'
' "40.000000" : 9792,\n'
' "50.000000" : 9792,\n'
' "60.000000" : 9792,\n'
' "70.000000" : 9792,\n'
' "80.000000" : 9792,\n'
' "90.000000" : 9920,\n'
' "95.000000" : 10048,\n'
' "99.000000" : 15040,\n'
' "99.500000" : 16192,\n'
' "99.900000" : 20096,\n'
' "99.950000" : 24960,\n'
' "99.990000" : 58112\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 8586,\n'
' "max" : 73798,\n'
' "mean" : 13064.128038,\n'
' "stddev" : '
'1183.145219,\n'
' "N" : 513818\n'
' },\n'
' "bw_min" : 36599,\n'
' "bw_max" : 36839,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'36707.785714,\n'
' "bw_dev" : 78.708985,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 73198,\n'
' "iops_max" : 73678,\n'
' "iops_mean" : '
'73415.428571,\n'
' "iops_stddev" : '
'157.295271,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.328571,\n'
' "sys_cpu" : 47.942857,\n'
' "ctx" : 513642,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010120,\n'
' "4" : 0.000000,\n'
' "10" : 94.480731,\n'
' "20" : 5.399772,\n'
' "50" : 0.087969,\n'
' "100" : 0.019657,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140012: Wed '
'Oct 26 04:29:30 2022\n'
' read: IOPS=73.4k, BW=35.8MiB/s '
'(37.6MB/s)(251MiB/7001msec)\n'
' slat (nsec): min=2856, '
'max=24596, avg=3174.84, '
'stdev=192.70\n'
' clat (nsec): min=942, '
'max=70743, avg=9889.29, '
'stdev=1172.70\n'
' lat (nsec): min=8586, '
'max=73798, avg=13064.13, '
'stdev=1183.15\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8512], 5.00th=[ '
'9536], 10.00th=[ 9664], 20.00th=[ '
'9664],\n'
' | 30.00th=[ 9792], 40.00th=[ '
'9792], 50.00th=[ 9792], 60.00th=[ '
'9792],\n'
' | 70.00th=[ 9792], 80.00th=[ '
'9792], 90.00th=[ 9920], '
'95.00th=[10048],\n'
' | 99.00th=[15040], '
'99.50th=[16192], 99.90th=[20096], '
'99.95th=[24960],\n'
' | 99.99th=[58112]\n'
' bw ( KiB/s): min=36599, '
'max=36839, per=100.00%, '
'avg=36707.79, stdev=78.71, '
'samples=14\n'
' iops : min=73198, '
'max=73678, avg=73415.43, '
'stdev=157.30, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'10=94.48%, 20=5.40%, 50=0.09%, '
'100=0.02%\n'
' cpu : usr=15.33%, '
'sys=47.94%, ctx=513642, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=513818,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=35.8MiB/s (37.6MB/s), '
'35.8MiB/s-35.8MiB/s '
'(37.6MB/s-37.6MB/s), io=251MiB '
'(263MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.output'),
'state': {'begin': 1666744281.6314862,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-xnvme_0.txt',
'cwd': 'None',
'elapsed': 0.7852046489715576,
'end': 1666744282.4166908,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.state')},
'cmd_94': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.output'),
'state': {'begin': 1666744283.4084666,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7857820987701416,
'end': 1666744284.1942487,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.state')},
'cmd_95': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.output'),
'state': {'begin': 1666744284.1963353,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-xnvme_1.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.09076476097107,
'end': 1666744295.2871,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.state')},
'cmd_96': {'output': 'libaio-xnvme: (g=0): rw=randread, '
'bs=(R) 512B-512B, (W) 512B-512B, '
'(T) 512B-512B, ioengine=xnvme, '
'iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751384,\n'
' "timestamp_ms" : '
'1666751384484,\n'
' "time" : "Wed Oct 26 04:29:44 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : "libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 345976320,\n'
' "io_kbytes" : 337867,\n'
' "bw_bytes" : 49418128,\n'
' "bw" : 48259,\n'
' "iops" : 96519.782888,\n'
' "runtime" : 7001,\n'
' "total_ios" : 675735,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2174,\n'
' "max" : 24436,\n'
' "mean" : 2431.272401,\n'
' "stddev" : 151.964015,\n'
' "N" : 675735\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 701,\n'
' "max" : 75923,\n'
' "mean" : 7477.687172,\n'
' "stddev" : '
'1131.118569,\n'
' "N" : 675735,\n'
' "percentile" : {\n'
' "1.000000" : 7136,\n'
' "5.000000" : 7200,\n'
' "10.000000" : 7200,\n'
' "20.000000" : 7264,\n'
' "30.000000" : 7264,\n'
' "40.000000" : 7264,\n'
' "50.000000" : 7328,\n'
' "60.000000" : 7328,\n'
' "70.000000" : 7392,\n'
' "80.000000" : 7392,\n'
' "90.000000" : 7584,\n'
' "95.000000" : 7712,\n'
' "99.000000" : 14016,\n'
' "99.500000" : 15040,\n'
' "99.900000" : 16064,\n'
' "99.950000" : 17024,\n'
' "99.990000" : 56064\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 7504,\n'
' "max" : 78317,\n'
' "mean" : 9908.959573,\n'
' "stddev" : '
'1141.464837,\n'
' "N" : 675735\n'
' },\n'
' "bw_min" : 48137,\n'
' "bw_max" : 48485,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'48272.857143,\n'
' "bw_dev" : 96.448358,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 96274,\n'
' "iops_max" : 96970,\n'
' "iops_mean" : '
'96545.785714,\n'
' "iops_stddev" : '
'192.890634,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 16.128571,\n'
' "sys_cpu" : 46.542857,\n'
' "ctx" : 675583,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.010000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.010000,\n'
' "10" : 98.266776,\n'
' "20" : 1.699483,\n'
' "50" : 0.011839,\n'
' "100" : 0.013023,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140052: Wed '
'Oct 26 04:29:44 2022\n'
' read: IOPS=96.5k, BW=47.1MiB/s '
'(49.4MB/s)(330MiB/7001msec)\n'
' slat (nsec): min=2174, '
'max=24436, avg=2431.27, '
'stdev=151.96\n'
' clat (nsec): min=701, '
'max=75923, avg=7477.69, '
'stdev=1131.12\n'
' lat (nsec): min=7504, '
'max=78317, avg=9908.96, '
'stdev=1141.46\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 7136], 5.00th=[ '
'7200], 10.00th=[ 7200], 20.00th=[ '
'7264],\n'
' | 30.00th=[ 7264], 40.00th=[ '
'7264], 50.00th=[ 7328], 60.00th=[ '
'7328],\n'
' | 70.00th=[ 7392], 80.00th=[ '
'7392], 90.00th=[ 7584], 95.00th=[ '
'7712],\n'
' | 99.00th=[14016], '
'99.50th=[15040], 99.90th=[16064], '
'99.95th=[17024],\n'
' | 99.99th=[56064]\n'
' bw ( KiB/s): min=48137, '
'max=48485, per=100.00%, '
'avg=48272.86, stdev=96.45, '
'samples=14\n'
' iops : min=96274, '
'max=96970, avg=96545.79, '
'stdev=192.89, samples=14\n'
' lat (nsec) : 750=0.01%, '
'1000=0.01%\n'
' lat (usec) : 4=0.01%, '
'10=98.27%, 20=1.70%, 50=0.01%, '
'100=0.01%\n'
' cpu : usr=16.13%, '
'sys=46.54%, ctx=675583, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=675735,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=47.1MiB/s (49.4MB/s), '
'47.1MiB/s-47.1MiB/s '
'(49.4MB/s-49.4MB/s), io=330MiB '
'(346MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.output'),
'state': {'begin': 1666744295.290078,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-xnvme_1.txt',
'cwd': 'None',
'elapsed': 0.7871439456939697,
'end': 1666744296.0772219,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.state')},
'cmd_97': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.output'),
'state': {'begin': 1666744297.0662894,
'cmd': 'rm '
'/tmp/fio-output_512_1_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7880120277404785,
'end': 1666744297.8543015,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.state')},
'cmd_98': {'output': '',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.output'),
'state': {'begin': 1666744297.8563988,
'cmd': '/opt/fio/bin/fio '
'--name="libaio-xnvme" '
'--rw="randread" '
'--size="12G" --iodepth="1" '
'--bs="512" --direct="1" '
'--thread="1" '
'--time_based="1" '
'--runtime="7" '
'--ramp_time="3" '
'--norandommap="1" '
'--allow_file_create="0" '
'--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
'--xnvme_be="linux" '
'--xnvme_async="libaio" '
'--xnvme_sync="nvme" '
'--xnvme_admin="nvme" '
'--xnvme_dev_nsid="1" '
'--filename="/dev/nvme0n1" '
'--output-format="normal,json" '
'--output="/tmp/fio-output_512_1_libaio-xnvme_2.txt" '
'--cpus_allowed="1"',
'cwd': 'None',
'elapsed': 11.089160203933716,
'end': 1666744308.945559,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.state')},
'cmd_99': {'output': 'libaio-xnvme: (g=0): rw=randread, '
'bs=(R) 512B-512B, (W) 512B-512B, '
'(T) 512B-512B, ioengine=xnvme, '
'iodepth=1\n'
'fio-3.32-dirty\n'
'Starting 1 thread\n'
'{\n'
' "fio version" : '
'"fio-3.32-dirty",\n'
' "timestamp" : 1666751398,\n'
' "timestamp_ms" : '
'1666751398140,\n'
' "time" : "Wed Oct 26 04:29:58 '
'2022",\n'
' "jobs" : [\n'
' {\n'
' "jobname" : '
'"libaio-xnvme",\n'
' "groupid" : 0,\n'
' "error" : 0,\n'
' "eta" : 0,\n'
' "elapsed" : 11,\n'
' "job options" : {\n'
' "name" : "libaio-xnvme",\n'
' "rw" : "randread",\n'
' "size" : "12G",\n'
' "iodepth" : "1",\n'
' "bs" : "512",\n'
' "direct" : "1",\n'
' "thread" : "1",\n'
' "time_based" : "1",\n'
' "runtime" : "7",\n'
' "ramp_time" : "3",\n'
' "norandommap" : "1",\n'
' "allow_file_create" : '
'"0",\n'
' "ioengine" : '
'"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
' "xnvme_be" : "linux",\n'
' "xnvme_async" : '
'"libaio",\n'
' "xnvme_sync" : "nvme",\n'
' "xnvme_admin" : "nvme",\n'
' "xnvme_dev_nsid" : "1",\n'
' "filename" : '
'"/dev/nvme0n1",\n'
' "cpus_allowed" : "1"\n'
' },\n'
' "read" : {\n'
' "io_bytes" : 264449536,\n'
' "io_kbytes" : 258251,\n'
' "bw_bytes" : 37773108,\n'
' "bw" : 36887,\n'
' "iops" : 73775.460648,\n'
' "runtime" : 7001,\n'
' "total_ios" : 516502,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 2955,\n'
' "max" : 35957,\n'
' "mean" : 3190.098706,\n'
' "stddev" : 227.777047,\n'
' "N" : 516503\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 892,\n'
' "max" : 117179,\n'
' "mean" : 9770.164466,\n'
' "stddev" : '
'1173.998407,\n'
' "N" : 516502,\n'
' "percentile" : {\n'
' "1.000000" : 8256,\n'
' "5.000000" : 9408,\n'
' "10.000000" : 9536,\n'
' "20.000000" : 9536,\n'
' "30.000000" : 9664,\n'
' "40.000000" : 9664,\n'
' "50.000000" : 9664,\n'
' "60.000000" : 9664,\n'
' "70.000000" : 9664,\n'
' "80.000000" : 9792,\n'
' "90.000000" : 9792,\n'
' "95.000000" : 9920,\n'
' "99.000000" : 14912,\n'
' "99.500000" : 16064,\n'
' "99.900000" : 20096,\n'
' "99.950000" : 25216,\n'
' "99.990000" : 55552\n'
' }\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 9898,\n'
' "max" : 120415,\n'
' "mean" : 12960.263335,\n'
' "stddev" : '
'1193.805394,\n'
' "N" : 516502\n'
' },\n'
' "bw_min" : 36783,\n'
' "bw_max" : 37012,\n'
' "bw_agg" : 100.000000,\n'
' "bw_mean" : '
'36898.714286,\n'
' "bw_dev" : 75.942720,\n'
' "bw_samples" : 14,\n'
' "iops_min" : 73566,\n'
' "iops_max" : 74024,\n'
' "iops_mean" : '
'73797.714286,\n'
' "iops_stddev" : '
'151.591774,\n'
' "iops_samples" : 14\n'
' },\n'
' "write" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "trim" : {\n'
' "io_bytes" : 0,\n'
' "io_kbytes" : 0,\n'
' "bw_bytes" : 0,\n'
' "bw" : 0,\n'
' "iops" : 0.000000,\n'
' "runtime" : 0,\n'
' "total_ios" : 0,\n'
' "short_ios" : 0,\n'
' "drop_ios" : 0,\n'
' "slat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "clat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' },\n'
' "bw_min" : 0,\n'
' "bw_max" : 0,\n'
' "bw_agg" : 0.000000,\n'
' "bw_mean" : 0.000000,\n'
' "bw_dev" : 0.000000,\n'
' "bw_samples" : 0,\n'
' "iops_min" : 0,\n'
' "iops_max" : 0,\n'
' "iops_mean" : 0.000000,\n'
' "iops_stddev" : '
'0.000000,\n'
' "iops_samples" : 0\n'
' },\n'
' "sync" : {\n'
' "total_ios" : 0,\n'
' "lat_ns" : {\n'
' "min" : 0,\n'
' "max" : 0,\n'
' "mean" : 0.000000,\n'
' "stddev" : 0.000000,\n'
' "N" : 0\n'
' }\n'
' },\n'
' "job_runtime" : 7000,\n'
' "usr_cpu" : 15.985714,\n'
' "sys_cpu" : 47.085714,\n'
' "ctx" : 516350,\n'
' "majf" : 0,\n'
' "minf" : 0,\n'
' "iodepth_level" : {\n'
' "1" : 100.000000,\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_submit" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "iodepth_complete" : {\n'
' "0" : 0.000000,\n'
' "4" : 100.000000,\n'
' "8" : 0.000000,\n'
' "16" : 0.000000,\n'
' "32" : 0.000000,\n'
' "64" : 0.000000,\n'
' ">=64" : 0.000000\n'
' },\n'
' "latency_ns" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.010000\n'
' },\n'
' "latency_us" : {\n'
' "2" : 0.010000,\n'
' "4" : 0.000000,\n'
' "10" : 95.703211,\n'
' "20" : 4.180429,\n'
' "50" : 0.089061,\n'
' "100" : 0.017812,\n'
' "250" : 0.010000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000\n'
' },\n'
' "latency_ms" : {\n'
' "2" : 0.000000,\n'
' "4" : 0.000000,\n'
' "10" : 0.000000,\n'
' "20" : 0.000000,\n'
' "50" : 0.000000,\n'
' "100" : 0.000000,\n'
' "250" : 0.000000,\n'
' "500" : 0.000000,\n'
' "750" : 0.000000,\n'
' "1000" : 0.000000,\n'
' "2000" : 0.000000,\n'
' ">=2000" : 0.000000\n'
' },\n'
' "latency_depth" : 1,\n'
' "latency_target" : 0,\n'
' "latency_percentile" : '
'100.000000,\n'
' "latency_window" : 0\n'
' }\n'
' ]\n'
'}\n'
'\n'
'libaio-xnvme: (groupid=0, '
'jobs=1): err= 0: pid=140092: Wed '
'Oct 26 04:29:58 2022\n'
' read: IOPS=73.8k, BW=36.0MiB/s '
'(37.8MB/s)(252MiB/7001msec)\n'
' slat (nsec): min=2955, '
'max=35957, avg=3190.10, '
'stdev=227.78\n'
' clat (nsec): min=892, '
'max=117179, avg=9770.16, '
'stdev=1174.00\n'
' lat (usec): min=9, max=120, '
'avg=12.96, stdev= 1.19\n'
' clat percentiles (nsec):\n'
' | 1.00th=[ 8256], 5.00th=[ '
'9408], 10.00th=[ 9536], 20.00th=[ '
'9536],\n'
' | 30.00th=[ 9664], 40.00th=[ '
'9664], 50.00th=[ 9664], 60.00th=[ '
'9664],\n'
' | 70.00th=[ 9664], 80.00th=[ '
'9792], 90.00th=[ 9792], 95.00th=[ '
'9920],\n'
' | 99.00th=[14912], '
'99.50th=[16064], 99.90th=[20096], '
'99.95th=[25216],\n'
' | 99.99th=[55552]\n'
' bw ( KiB/s): min=36783, '
'max=37012, per=100.00%, '
'avg=36898.71, stdev=75.94, '
'samples=14\n'
' iops : min=73566, '
'max=74024, avg=73797.71, '
'stdev=151.59, samples=14\n'
' lat (nsec) : 1000=0.01%\n'
' lat (usec) : 2=0.01%, '
'10=95.70%, 20=4.18%, 50=0.09%, '
'100=0.02%\n'
' lat (usec) : 250=0.01%\n'
' cpu : usr=15.99%, '
'sys=47.09%, ctx=516350, majf=0, '
'minf=0\n'
' IO depths : 1=100.0%, '
'2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, >=64=0.0%\n'
' submit : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' complete : 0=0.0%, '
'4=100.0%, 8=0.0%, 16=0.0%, '
'32=0.0%, 64=0.0%, >=64=0.0%\n'
' issued rwts: '
'total=516502,0,0,0 short=0,0,0,0 '
'dropped=0,0,0,0\n'
' latency : target=0, '
'window=0, percentile=100.00%, '
'depth=1\n'
'\n'
'Run status group 0 (all jobs):\n'
' READ: bw=36.0MiB/s (37.8MB/s), '
'36.0MiB/s-36.0MiB/s '
'(37.8MB/s-37.8MB/s), io=252MiB '
'(264MB), run=7001-7001msec\n',
'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.output'),
'state': {'begin': 1666744308.9486403,
'cmd': 'cat '
'/tmp/fio-output_512_1_libaio-xnvme_2.txt',
'cwd': 'None',
'elapsed': 0.7886214256286621,
'end': 1666744309.7372618,
'err': 0,
'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.output'},
'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.state')}}},
'id': '003_fio',
'name': 'fio',
'nr': 3,
'status': {'elapsed': 1891.1022799015045,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'fioe',
'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['512'], 'repetitions': 3}}
$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666750939,
"timestamp_ms" : 1666750939831,
"time" : "Wed Oct 26 04:22:19 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 298279936,
"io_kbytes" : 291289,
"bw_bytes" : 42605332,
"bw" : 41606,
"iops" : 83213.540923,
"runtime" : 7001,
"total_ios" : 582578,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2224,
"max" : 28664,
"mean" : 2408.062177,
"stddev" : 216.254296,
"N" : 582578
},
"clat_ns" : {
"min" : 171,
"max" : 108203,
"mean" : 9002.665866,
"stddev" : 1272.220914,
"N" : 582578,
"percentile" : {
"1.000000" : 8640,
"5.000000" : 8640,
"10.000000" : 8768,
"20.000000" : 8768,
"30.000000" : 8768,
"40.000000" : 8768,
"50.000000" : 8768,
"60.000000" : 8896,
"70.000000" : 8896,
"80.000000" : 8896,
"90.000000" : 9024,
"95.000000" : 9152,
"99.000000" : 15680,
"99.500000" : 16320,
"99.900000" : 19072,
"99.950000" : 21888,
"99.990000" : 55552
}
},
"lat_ns" : {
"min" : 8035,
"max" : 110557,
"mean" : 11410.728043,
"stddev" : 1289.083345,
"N" : 582578
},
"bw_min" : 41479,
"bw_max" : 41783,
"bw_agg" : 100.000000,
"bw_mean" : 41617.071429,
"bw_dev" : 75.011245,
"bw_samples" : 14,
"iops_min" : 82958,
"iops_max" : 83566,
"iops_mean" : 83234.142857,
"iops_stddev" : 150.022489,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.700000,
"sys_cpu" : 46.957143,
"ctx" : 582454,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.381638,
"20" : 2.541119,
"50" : 0.052868,
"100" : 0.013389,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=138675: Wed Oct 26 04:22:19 2022
read: IOPS=83.2k, BW=40.6MiB/s (42.6MB/s)(284MiB/7001msec)
slat (nsec): min=2224, max=28664, avg=2408.06, stdev=216.25
clat (nsec): min=171, max=108203, avg=9002.67, stdev=1272.22
lat (usec): min=8, max=110, avg=11.41, stdev= 1.29
clat percentiles (nsec):
| 1.00th=[ 8640], 5.00th=[ 8640], 10.00th=[ 8768], 20.00th=[ 8768],
| 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8896],
| 70.00th=[ 8896], 80.00th=[ 8896], 90.00th=[ 9024], 95.00th=[ 9152],
| 99.00th=[15680], 99.50th=[16320], 99.90th=[19072], 99.95th=[21888],
| 99.99th=[55552]
bw ( KiB/s): min=41479, max=41783, per=100.00%, avg=41617.07, stdev=75.01, samples=14
iops : min=82958, max=83566, avg=83234.14, stdev=150.02, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=97.38%, 20=2.54%, 50=0.05%, 100=0.01%
lat (usec) : 250=0.01%
cpu : usr=11.70%, sys=46.96%, ctx=582454, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=582578,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=40.6MiB/s (42.6MB/s), 40.6MiB/s-40.6MiB/s (42.6MB/s-42.6MB/s), io=284MiB (298MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666750953,
"timestamp_ms" : 1666750953459,
"time" : "Wed Oct 26 04:22:33 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 299215872,
"io_kbytes" : 292203,
"bw_bytes" : 42739018,
"bw" : 41737,
"iops" : 83474.646479,
"runtime" : 7001,
"total_ios" : 584406,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2245,
"max" : 30237,
"mean" : 2410.540881,
"stddev" : 223.499942,
"N" : 584406
},
"clat_ns" : {
"min" : 190,
"max" : 117380,
"mean" : 8968.711232,
"stddev" : 1341.037263,
"N" : 584406,
"percentile" : {
"1.000000" : 8640,
"5.000000" : 8640,
"10.000000" : 8640,
"20.000000" : 8768,
"30.000000" : 8768,
"40.000000" : 8768,
"50.000000" : 8768,
"60.000000" : 8768,
"70.000000" : 8768,
"80.000000" : 8896,
"90.000000" : 8896,
"95.000000" : 9024,
"99.000000" : 15680,
"99.500000" : 16512,
"99.900000" : 19072,
"99.950000" : 22912,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 8446,
"max" : 119784,
"mean" : 11379.252112,
"stddev" : 1357.748455,
"N" : 584406
},
"bw_min" : 41568,
"bw_max" : 41880,
"bw_agg" : 100.000000,
"bw_mean" : 41747.214286,
"bw_dev" : 89.871856,
"bw_samples" : 14,
"iops_min" : 83136,
"iops_max" : 83760,
"iops_mean" : 83494.500000,
"iops_stddev" : 179.757422,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.600000,
"sys_cpu" : 47.971429,
"ctx" : 584275,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.434147,
"20" : 2.479270,
"50" : 0.057152,
"100" : 0.018651,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=138713: Wed Oct 26 04:22:33 2022
read: IOPS=83.5k, BW=40.8MiB/s (42.7MB/s)(285MiB/7001msec)
slat (nsec): min=2245, max=30237, avg=2410.54, stdev=223.50
clat (nsec): min=190, max=117380, avg=8968.71, stdev=1341.04
lat (usec): min=8, max=119, avg=11.38, stdev= 1.36
clat percentiles (nsec):
| 1.00th=[ 8640], 5.00th=[ 8640], 10.00th=[ 8640], 20.00th=[ 8768],
| 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8768],
| 70.00th=[ 8768], 80.00th=[ 8896], 90.00th=[ 8896], 95.00th=[ 9024],
| 99.00th=[15680], 99.50th=[16512], 99.90th=[19072], 99.95th=[22912],
| 99.99th=[57088]
bw ( KiB/s): min=41568, max=41880, per=100.00%, avg=41747.21, stdev=89.87, samples=14
iops : min=83136, max=83760, avg=83494.50, stdev=179.76, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 10=97.43%, 20=2.48%, 50=0.06%, 100=0.02%, 250=0.01%
cpu : usr=10.60%, sys=47.97%, ctx=584275, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=584406,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=40.8MiB/s (42.7MB/s), 40.8MiB/s-40.8MiB/s (42.7MB/s-42.7MB/s), io=285MiB (299MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt': No such file or directory
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666750967,
"timestamp_ms" : 1666750967372,
"time" : "Wed Oct 26 04:22:47 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 299606016,
"io_kbytes" : 292584,
"bw_bytes" : 42794745,
"bw" : 41791,
"iops" : 83583.345236,
"runtime" : 7001,
"total_ios" : 585167,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2244,
"max" : 36078,
"mean" : 2458.829481,
"stddev" : 222.580918,
"N" : 585168
},
"clat_ns" : {
"min" : 170,
"max" : 108393,
"mean" : 8930.150774,
"stddev" : 1341.344004,
"N" : 585167,
"percentile" : {
"1.000000" : 8512,
"5.000000" : 8640,
"10.000000" : 8640,
"20.000000" : 8640,
"30.000000" : 8640,
"40.000000" : 8768,
"50.000000" : 8768,
"60.000000" : 8768,
"70.000000" : 8768,
"80.000000" : 8768,
"90.000000" : 8896,
"95.000000" : 9024,
"99.000000" : 15680,
"99.500000" : 16320,
"99.900000" : 19072,
"99.950000" : 22656,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 8526,
"max" : 110867,
"mean" : 11388.980330,
"stddev" : 1357.766479,
"N" : 585167
},
"bw_min" : 41636,
"bw_max" : 41936,
"bw_agg" : 100.000000,
"bw_mean" : 41802.000000,
"bw_dev" : 76.181968,
"bw_samples" : 14,
"iops_min" : 83272,
"iops_max" : 83872,
"iops_mean" : 83604.071429,
"iops_stddev" : 152.343975,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.771429,
"sys_cpu" : 45.685714,
"ctx" : 585039,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.435775,
"20" : 2.483394,
"50" : 0.051097,
"100" : 0.018285,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=138751: Wed Oct 26 04:22:47 2022
read: IOPS=83.6k, BW=40.8MiB/s (42.8MB/s)(286MiB/7001msec)
slat (nsec): min=2244, max=36078, avg=2458.83, stdev=222.58
clat (nsec): min=170, max=108393, avg=8930.15, stdev=1341.34
lat (usec): min=8, max=110, avg=11.39, stdev= 1.36
clat percentiles (nsec):
| 1.00th=[ 8512], 5.00th=[ 8640], 10.00th=[ 8640], 20.00th=[ 8640],
| 30.00th=[ 8640], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8768],
| 70.00th=[ 8768], 80.00th=[ 8768], 90.00th=[ 8896], 95.00th=[ 9024],
| 99.00th=[15680], 99.50th=[16320], 99.90th=[19072], 99.95th=[22656],
| 99.99th=[57088]
bw ( KiB/s): min=41636, max=41936, per=100.00%, avg=41802.00, stdev=76.18, samples=14
iops : min=83272, max=83872, avg=83604.07, stdev=152.34, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 10=97.44%, 20=2.48%, 50=0.05%, 100=0.02%, 250=0.01%
cpu : usr=12.77%, sys=45.69%, ctx=585039, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=585167,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=40.8MiB/s (42.8MB/s), 40.8MiB/s-40.8MiB/s (42.8MB/s-42.8MB/s), io=286MiB (300MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt
$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751411,
"timestamp_ms" : 1666751411748,
"time" : "Wed Oct 26 04:30:11 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 737202688,
"io_kbytes" : 719924,
"bw_bytes" : 105299626,
"bw" : 102831,
"iops" : 205663.190973,
"runtime" : 7001,
"total_ios" : 1439848,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1152,
"max" : 27401,
"mean" : 1324.465338,
"stddev" : 150.281294,
"N" : 1439848
},
"clat_ns" : {
"min" : 80,
"max" : 101000,
"mean" : 8115.300295,
"stddev" : 2223.862930,
"N" : 1439849,
"percentile" : {
"1.000000" : 6048,
"5.000000" : 6112,
"10.000000" : 6112,
"20.000000" : 6176,
"30.000000" : 6240,
"40.000000" : 6304,
"50.000000" : 8768,
"60.000000" : 9536,
"70.000000" : 9536,
"80.000000" : 9664,
"90.000000" : 9664,
"95.000000" : 9792,
"99.000000" : 16512,
"99.500000" : 17280,
"99.900000" : 18560,
"99.950000" : 22656,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5911,
"max" : 102202,
"mean" : 9439.765673,
"stddev" : 2290.846561,
"N" : 1439849
},
"bw_min" : 102294,
"bw_max" : 103753,
"bw_agg" : 100.000000,
"bw_mean" : 102856.357143,
"bw_dev" : 489.366562,
"bw_samples" : 14,
"iops_min" : 204588,
"iops_max" : 207506,
"iops_mean" : 205712.714286,
"iops_stddev" : 978.733124,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.900000,
"sys_cpu" : 50.014286,
"ctx" : 719850,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 96.593182,
"20" : 3.315558,
"50" : 0.057784,
"100" : 0.023961,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=140132: Wed Oct 26 04:30:11 2022
read: IOPS=206k, BW=100MiB/s (105MB/s)(703MiB/7001msec)
slat (nsec): min=1152, max=27401, avg=1324.47, stdev=150.28
clat (nsec): min=80, max=101000, avg=8115.30, stdev=2223.86
lat (usec): min=5, max=102, avg= 9.44, stdev= 2.29
clat percentiles (nsec):
| 1.00th=[ 6048], 5.00th=[ 6112], 10.00th=[ 6112], 20.00th=[ 6176],
| 30.00th=[ 6240], 40.00th=[ 6304], 50.00th=[ 8768], 60.00th=[ 9536],
| 70.00th=[ 9536], 80.00th=[ 9664], 90.00th=[ 9664], 95.00th=[ 9792],
| 99.00th=[16512], 99.50th=[17280], 99.90th=[18560], 99.95th=[22656],
| 99.99th=[61184]
bw ( KiB/s): min=102294, max=103753, per=100.00%, avg=102856.36, stdev=489.37, samples=14
iops : min=204588, max=207506, avg=205712.71, stdev=978.73, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=96.59%, 20=3.32%, 50=0.06%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=18.90%, sys=50.01%, ctx=719850, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1439848,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=703MiB (737MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751425,
"timestamp_ms" : 1666751425348,
"time" : "Wed Oct 26 04:30:25 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 741398528,
"io_kbytes" : 724022,
"bw_bytes" : 105898947,
"bw" : 103416,
"iops" : 206833.738037,
"runtime" : 7001,
"total_ios" : 1448043,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1142,
"max" : 40395,
"mean" : 1297.229181,
"stddev" : 146.300890,
"N" : 1448043
},
"clat_ns" : {
"min" : 80,
"max" : 96732,
"mean" : 8087.446961,
"stddev" : 2216.390509,
"N" : 1448044,
"percentile" : {
"1.000000" : 5984,
"5.000000" : 6048,
"10.000000" : 6112,
"20.000000" : 6176,
"30.000000" : 6240,
"40.000000" : 6304,
"50.000000" : 8096,
"60.000000" : 9408,
"70.000000" : 9536,
"80.000000" : 9536,
"90.000000" : 9664,
"95.000000" : 9792,
"99.000000" : 16320,
"99.500000" : 17280,
"99.900000" : 19072,
"99.950000" : 26752,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 6071,
"max" : 97904,
"mean" : 9384.676166,
"stddev" : 2292.860984,
"N" : 1448044
},
"bw_min" : 103177,
"bw_max" : 103737,
"bw_agg" : 100.000000,
"bw_mean" : 103442.142857,
"bw_dev" : 188.649801,
"bw_samples" : 14,
"iops_min" : 206354,
"iops_max" : 207474,
"iops_mean" : 206884.285714,
"iops_stddev" : 377.119330,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.357143,
"sys_cpu" : 51.185714,
"ctx" : 723965,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 96.716948,
"20" : 3.182640,
"50" : 0.069128,
"100" : 0.023135,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=140171: Wed Oct 26 04:30:25 2022
read: IOPS=207k, BW=101MiB/s (106MB/s)(707MiB/7001msec)
slat (nsec): min=1142, max=40395, avg=1297.23, stdev=146.30
clat (nsec): min=80, max=96732, avg=8087.45, stdev=2216.39
lat (nsec): min=6071, max=97904, avg=9384.68, stdev=2292.86
clat percentiles (nsec):
| 1.00th=[ 5984], 5.00th=[ 6048], 10.00th=[ 6112], 20.00th=[ 6176],
| 30.00th=[ 6240], 40.00th=[ 6304], 50.00th=[ 8096], 60.00th=[ 9408],
| 70.00th=[ 9536], 80.00th=[ 9536], 90.00th=[ 9664], 95.00th=[ 9792],
| 99.00th=[16320], 99.50th=[17280], 99.90th=[19072], 99.95th=[26752],
| 99.99th=[61184]
bw ( KiB/s): min=103177, max=103737, per=100.00%, avg=103442.14, stdev=188.65, samples=14
iops : min=206354, max=207474, avg=206884.29, stdev=377.12, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=96.72%, 20=3.18%, 50=0.07%
lat (usec) : 100=0.02%
cpu : usr=17.36%, sys=51.19%, ctx=723965, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1448043,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=101MiB/s (106MB/s), 101MiB/s-101MiB/s (106MB/s-106MB/s), io=707MiB (741MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751438,
"timestamp_ms" : 1666751438950,
"time" : "Wed Oct 26 04:30:38 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 734948864,
"io_kbytes" : 717723,
"bw_bytes" : 104977698,
"bw" : 102517,
"iops" : 205034.280817,
"runtime" : 7001,
"total_ios" : 1435445,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1182,
"max" : 27221,
"mean" : 1330.533098,
"stddev" : 132.496465,
"N" : 1435446
},
"clat_ns" : {
"min" : 100,
"max" : 117019,
"mean" : 8150.587585,
"stddev" : 2236.609171,
"N" : 1435446,
"percentile" : {
"1.000000" : 6048,
"5.000000" : 6112,
"10.000000" : 6112,
"20.000000" : 6240,
"30.000000" : 6304,
"40.000000" : 6304,
"50.000000" : 8640,
"60.000000" : 9536,
"70.000000" : 9664,
"80.000000" : 9664,
"90.000000" : 9792,
"95.000000" : 9792,
"99.000000" : 16512,
"99.500000" : 17280,
"99.900000" : 20096,
"99.950000" : 27008,
"99.990000" : 62208
}
},
"lat_ns" : {
"min" : 5871,
"max" : 118241,
"mean" : 9481.120683,
"stddev" : 2308.727929,
"N" : 1435446
},
"bw_min" : 102325,
"bw_max" : 102710,
"bw_agg" : 100.000000,
"bw_mean" : 102550.214286,
"bw_dev" : 111.494173,
"bw_samples" : 14,
"iops_min" : 204650,
"iops_max" : 205420,
"iops_mean" : 205100.428571,
"iops_stddev" : 223.226933,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.157143,
"sys_cpu" : 48.828571,
"ctx" : 717624,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 96.261577,
"20" : 3.628561,
"50" : 0.077885,
"100" : 0.021875,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=140209: Wed Oct 26 04:30:38 2022
read: IOPS=205k, BW=100MiB/s (105MB/s)(701MiB/7001msec)
slat (nsec): min=1182, max=27221, avg=1330.53, stdev=132.50
clat (nsec): min=100, max=117019, avg=8150.59, stdev=2236.61
lat (usec): min=5, max=118, avg= 9.48, stdev= 2.31
clat percentiles (nsec):
| 1.00th=[ 6048], 5.00th=[ 6112], 10.00th=[ 6112], 20.00th=[ 6240],
| 30.00th=[ 6304], 40.00th=[ 6304], 50.00th=[ 8640], 60.00th=[ 9536],
| 70.00th=[ 9664], 80.00th=[ 9664], 90.00th=[ 9792], 95.00th=[ 9792],
| 99.00th=[16512], 99.50th=[17280], 99.90th=[20096], 99.95th=[27008],
| 99.99th=[62208]
bw ( KiB/s): min=102325, max=102710, per=100.00%, avg=102550.21, stdev=111.49, samples=14
iops : min=204650, max=205420, avg=205100.43, stdev=223.23, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=96.26%, 20=3.63%, 50=0.08%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=20.16%, sys=48.83%, ctx=717624, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1435445,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=701MiB (735MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751452,
"timestamp_ms" : 1666751452581,
"time" : "Wed Oct 26 04:30:52 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 726595072,
"io_kbytes" : 709565,
"bw_bytes" : 103784469,
"bw" : 101352,
"iops" : 202703.756606,
"runtime" : 7001,
"total_ios" : 1419129,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1212,
"max" : 64080,
"mean" : 1388.671381,
"stddev" : 173.652155,
"N" : 1419130
},
"clat_ns" : {
"min" : 121,
"max" : 84979,
"mean" : 8206.406104,
"stddev" : 2089.729687,
"N" : 1419130,
"percentile" : {
"1.000000" : 5216,
"5.000000" : 5536,
"10.000000" : 6304,
"20.000000" : 6368,
"30.000000" : 6496,
"40.000000" : 7776,
"50.000000" : 8096,
"60.000000" : 8256,
"70.000000" : 9792,
"80.000000" : 9920,
"90.000000" : 10048,
"95.000000" : 10176,
"99.000000" : 16512,
"99.500000" : 17024,
"99.900000" : 18560,
"99.950000" : 22912,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5921,
"max" : 86271,
"mean" : 9595.077485,
"stddev" : 2157.718644,
"N" : 1419130
},
"bw_min" : 101000,
"bw_max" : 101608,
"bw_agg" : 100.000000,
"bw_mean" : 101386.571429,
"bw_dev" : 192.066380,
"bw_samples" : 14,
"iops_min" : 202000,
"iops_max" : 203216,
"iops_mean" : 202773.285714,
"iops_stddev" : 384.198953,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.300000,
"sys_cpu" : 49.800000,
"ctx" : 709604,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 87.470413,
"20" : 12.438193,
"50" : 0.058275,
"100" : 0.019730,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=140247: Wed Oct 26 04:30:52 2022
read: IOPS=203k, BW=99.0MiB/s (104MB/s)(693MiB/7001msec)
slat (nsec): min=1212, max=64080, avg=1388.67, stdev=173.65
clat (nsec): min=121, max=84979, avg=8206.41, stdev=2089.73
lat (nsec): min=5921, max=86271, avg=9595.08, stdev=2157.72
clat percentiles (nsec):
| 1.00th=[ 5216], 5.00th=[ 5536], 10.00th=[ 6304], 20.00th=[ 6368],
| 30.00th=[ 6496], 40.00th=[ 7776], 50.00th=[ 8096], 60.00th=[ 8256],
| 70.00th=[ 9792], 80.00th=[ 9920], 90.00th=[10048], 95.00th=[10176],
| 99.00th=[16512], 99.50th=[17024], 99.90th=[18560], 99.95th=[22912],
| 99.99th=[61184]
bw ( KiB/s): min=101000, max=101608, per=100.00%, avg=101386.57, stdev=192.07, samples=14
iops : min=202000, max=203216, avg=202773.29, stdev=384.20, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=87.47%, 20=12.44%, 50=0.06%
lat (usec) : 100=0.02%
cpu : usr=19.30%, sys=49.80%, ctx=709604, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1419129,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=99.0MiB/s (104MB/s), 99.0MiB/s-99.0MiB/s (104MB/s-104MB/s), io=693MiB (727MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751466,
"timestamp_ms" : 1666751466213,
"time" : "Wed Oct 26 04:31:06 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 722791424,
"io_kbytes" : 705851,
"bw_bytes" : 103241168,
"bw" : 100821,
"iops" : 201642.765319,
"runtime" : 7001,
"total_ios" : 1411701,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1202,
"max" : 25348,
"mean" : 1368.416727,
"stddev" : 146.449095,
"N" : 1411701
},
"clat_ns" : {
"min" : 120,
"max" : 109946,
"mean" : 8258.915002,
"stddev" : 2093.679574,
"N" : 1411702,
"percentile" : {
"1.000000" : 5472,
"5.000000" : 6240,
"10.000000" : 6240,
"20.000000" : 6304,
"30.000000" : 6432,
"40.000000" : 7776,
"50.000000" : 8032,
"60.000000" : 8256,
"70.000000" : 9792,
"80.000000" : 9920,
"90.000000" : 9920,
"95.000000" : 10048,
"99.000000" : 16512,
"99.500000" : 17024,
"99.900000" : 18560,
"99.950000" : 23680,
"99.990000" : 60672
}
},
"lat_ns" : {
"min" : 6092,
"max" : 111208,
"mean" : 9627.331782,
"stddev" : 2159.028715,
"N" : 1411702
},
"bw_min" : 100667,
"bw_max" : 101169,
"bw_agg" : 100.000000,
"bw_mean" : 100850.000000,
"bw_dev" : 142.687394,
"bw_samples" : 14,
"iops_min" : 201334,
"iops_max" : 202338,
"iops_mean" : 201700.000000,
"iops_stddev" : 285.374787,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 21.028571,
"sys_cpu" : 47.785714,
"ctx" : 705798,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 91.422546,
"20" : 8.482674,
"50" : 0.060565,
"100" : 0.022597,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=140286: Wed Oct 26 04:31:06 2022
read: IOPS=202k, BW=98.5MiB/s (103MB/s)(689MiB/7001msec)
slat (nsec): min=1202, max=25348, avg=1368.42, stdev=146.45
clat (nsec): min=120, max=109946, avg=8258.92, stdev=2093.68
lat (usec): min=6, max=111, avg= 9.63, stdev= 2.16
clat percentiles (nsec):
| 1.00th=[ 5472], 5.00th=[ 6240], 10.00th=[ 6240], 20.00th=[ 6304],
| 30.00th=[ 6432], 40.00th=[ 7776], 50.00th=[ 8032], 60.00th=[ 8256],
| 70.00th=[ 9792], 80.00th=[ 9920], 90.00th=[ 9920], 95.00th=[10048],
| 99.00th=[16512], 99.50th=[17024], 99.90th=[18560], 99.95th=[23680],
| 99.99th=[60672]
bw ( KiB/s): min=100667, max=101169, per=100.00%, avg=100850.00, stdev=142.69, samples=14
iops : min=201334, max=202338, avg=201700.00, stdev=285.37, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=91.42%, 20=8.48%, 50=0.06%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=21.03%, sys=47.79%, ctx=705798, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1411701,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=98.5MiB/s (103MB/s), 98.5MiB/s-98.5MiB/s (103MB/s-103MB/s), io=689MiB (723MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751479,
"timestamp_ms" : 1666751479857,
"time" : "Wed Oct 26 04:31:19 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 727922688,
"io_kbytes" : 710862,
"bw_bytes" : 103974101,
"bw" : 101537,
"iops" : 203074.275104,
"runtime" : 7001,
"total_ios" : 1421723,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1212,
"max" : 24436,
"mean" : 1384.957250,
"stddev" : 152.943213,
"N" : 1421723
},
"clat_ns" : {
"min" : 120,
"max" : 77255,
"mean" : 8203.277642,
"stddev" : 2113.140529,
"N" : 1421724,
"percentile" : {
"1.000000" : 5280,
"5.000000" : 6176,
"10.000000" : 6240,
"20.000000" : 6304,
"30.000000" : 6432,
"40.000000" : 7776,
"50.000000" : 8032,
"60.000000" : 8256,
"70.000000" : 9792,
"80.000000" : 9920,
"90.000000" : 9920,
"95.000000" : 10048,
"99.000000" : 16320,
"99.500000" : 17280,
"99.900000" : 18816,
"99.950000" : 23424,
"99.990000" : 60672
}
},
"lat_ns" : {
"min" : 6181,
"max" : 78517,
"mean" : 9588.234953,
"stddev" : 2180.631735,
"N" : 1421724
},
"bw_min" : 101351,
"bw_max" : 101836,
"bw_agg" : 100.000000,
"bw_mean" : 101561.714286,
"bw_dev" : 140.942666,
"bw_samples" : 14,
"iops_min" : 202702,
"iops_max" : 203672,
"iops_mean" : 203123.428571,
"iops_stddev" : 281.885332,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.828571,
"sys_cpu" : 49.057143,
"ctx" : 710806,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 92.513310,
"20" : 7.385335,
"50" : 0.065695,
"100" : 0.023774,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=140325: Wed Oct 26 04:31:19 2022
read: IOPS=203k, BW=99.2MiB/s (104MB/s)(694MiB/7001msec)
slat (nsec): min=1212, max=24436, avg=1384.96, stdev=152.94
clat (nsec): min=120, max=77255, avg=8203.28, stdev=2113.14
lat (nsec): min=6181, max=78517, avg=9588.23, stdev=2180.63
clat percentiles (nsec):
| 1.00th=[ 5280], 5.00th=[ 6176], 10.00th=[ 6240], 20.00th=[ 6304],
| 30.00th=[ 6432], 40.00th=[ 7776], 50.00th=[ 8032], 60.00th=[ 8256],
| 70.00th=[ 9792], 80.00th=[ 9920], 90.00th=[ 9920], 95.00th=[10048],
| 99.00th=[16320], 99.50th=[17280], 99.90th=[18816], 99.95th=[23424],
| 99.99th=[60672]
bw ( KiB/s): min=101351, max=101836, per=100.00%, avg=101561.71, stdev=140.94, samples=14
iops : min=202702, max=203672, avg=203123.43, stdev=281.89, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=92.51%, 20=7.39%, 50=0.07%
lat (usec) : 100=0.02%
cpu : usr=19.83%, sys=49.06%, ctx=710806, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1421723,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=99.2MiB/s (104MB/s), 99.2MiB/s-99.2MiB/s (104MB/s-104MB/s), io=694MiB (728MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666750981,
"timestamp_ms" : 1666750981040,
"time" : "Wed Oct 26 04:23:01 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 290464256,
"io_kbytes" : 283656,
"bw_bytes" : 41488966,
"bw" : 40516,
"iops" : 81032.995286,
"runtime" : 7001,
"total_ios" : 567312,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2304,
"max" : 40236,
"mean" : 2576.996690,
"stddev" : 232.578518,
"N" : 567313
},
"clat_ns" : {
"min" : 340,
"max" : 70823,
"mean" : 9183.791792,
"stddev" : 1245.876786,
"N" : 567312,
"percentile" : {
"1.000000" : 8768,
"5.000000" : 8896,
"10.000000" : 8896,
"20.000000" : 8896,
"30.000000" : 8896,
"40.000000" : 9024,
"50.000000" : 9024,
"60.000000" : 9024,
"70.000000" : 9024,
"80.000000" : 9024,
"90.000000" : 9152,
"95.000000" : 9280,
"99.000000" : 15552,
"99.500000" : 16512,
"99.900000" : 19328,
"99.950000" : 21632,
"99.990000" : 55552
}
},
"lat_ns" : {
"min" : 7985,
"max" : 73398,
"mean" : 11760.788467,
"stddev" : 1263.053401,
"N" : 567312
},
"bw_min" : 40426,
"bw_max" : 40799,
"bw_agg" : 100.000000,
"bw_mean" : 40525.857143,
"bw_dev" : 100.937038,
"bw_samples" : 14,
"iops_min" : 80852,
"iops_max" : 81598,
"iops_mean" : 81051.714286,
"iops_stddev" : 201.874077,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.071429,
"sys_cpu" : 45.314286,
"ctx" : 567166,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.013220,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.317525,
"20" : 2.597512,
"50" : 0.056583,
"100" : 0.014983,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=138790: Wed Oct 26 04:23:01 2022
read: IOPS=81.0k, BW=39.6MiB/s (41.5MB/s)(277MiB/7001msec)
slat (nsec): min=2304, max=40236, avg=2577.00, stdev=232.58
clat (nsec): min=340, max=70823, avg=9183.79, stdev=1245.88
lat (nsec): min=7985, max=73398, avg=11760.79, stdev=1263.05
clat percentiles (nsec):
| 1.00th=[ 8768], 5.00th=[ 8896], 10.00th=[ 8896], 20.00th=[ 8896],
| 30.00th=[ 8896], 40.00th=[ 9024], 50.00th=[ 9024], 60.00th=[ 9024],
| 70.00th=[ 9024], 80.00th=[ 9024], 90.00th=[ 9152], 95.00th=[ 9280],
| 99.00th=[15552], 99.50th=[16512], 99.90th=[19328], 99.95th=[21632],
| 99.99th=[55552]
bw ( KiB/s): min=40426, max=40799, per=100.00%, avg=40525.86, stdev=100.94, samples=14
iops : min=80852, max=81598, avg=81051.71, stdev=201.87, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 10=97.32%, 20=2.60%, 50=0.06%, 100=0.01%
cpu : usr=14.07%, sys=45.31%, ctx=567166, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=567312,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=39.6MiB/s (41.5MB/s), 39.6MiB/s-39.6MiB/s (41.5MB/s-41.5MB/s), io=277MiB (290MB), run=7001-7001msec
$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751494,
"timestamp_ms" : 1666751494875,
"time" : "Wed Oct 26 04:31:34 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 840442368,
"io_kbytes" : 820744,
"bw_bytes" : 120046045,
"bw" : 117232,
"iops" : 234464.790744,
"runtime" : 7001,
"total_ios" : 1641488,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 230,
"max" : 20989,
"mean" : 273.106698,
"stddev" : 42.767660,
"N" : 1641488
},
"clat_ns" : {
"min" : 270,
"max" : 112260,
"mean" : 8033.977383,
"stddev" : 2024.239647,
"N" : 1641489,
"percentile" : {
"1.000000" : 6368,
"5.000000" : 6432,
"10.000000" : 6496,
"20.000000" : 6944,
"30.000000" : 7392,
"40.000000" : 7456,
"50.000000" : 7648,
"60.000000" : 7840,
"70.000000" : 8640,
"80.000000" : 8768,
"90.000000" : 8896,
"95.000000" : 9024,
"99.000000" : 18816,
"99.500000" : 20096,
"99.900000" : 22400,
"99.950000" : 27776,
"99.990000" : 62208
}
},
"lat_ns" : {
"min" : 6081,
"max" : 112541,
"mean" : 8307.084073,
"stddev" : 2025.511582,
"N" : 1641489
},
"bw_min" : 116935,
"bw_max" : 117650,
"bw_agg" : 100.000000,
"bw_mean" : 117259.714286,
"bw_dev" : 195.708350,
"bw_samples" : 14,
"iops_min" : 233870,
"iops_max" : 235300,
"iops_mean" : 234519.428571,
"iops_stddev" : 391.416700,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 25,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.981480,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 96.879478,
"20" : 2.601055,
"50" : 0.497293,
"100" : 0.021505,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140374: Wed Oct 26 04:31:34 2022
read: IOPS=234k, BW=114MiB/s (120MB/s)(802MiB/7001msec)
slat (nsec): min=230, max=20989, avg=273.11, stdev=42.77
clat (nsec): min=270, max=112260, avg=8033.98, stdev=2024.24
lat (usec): min=6, max=112, avg= 8.31, stdev= 2.03
clat percentiles (nsec):
| 1.00th=[ 6368], 5.00th=[ 6432], 10.00th=[ 6496], 20.00th=[ 6944],
| 30.00th=[ 7392], 40.00th=[ 7456], 50.00th=[ 7648], 60.00th=[ 7840],
| 70.00th=[ 8640], 80.00th=[ 8768], 90.00th=[ 8896], 95.00th=[ 9024],
| 99.00th=[18816], 99.50th=[20096], 99.90th=[22400], 99.95th=[27776],
| 99.99th=[62208]
bw ( KiB/s): min=116935, max=117650, per=100.00%, avg=117259.71, stdev=195.71, samples=14
iops : min=233870, max=235300, avg=234519.43, stdev=391.42, samples=14
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=96.88%, 20=2.60%, 50=0.50%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=25, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1641488,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=114MiB/s (120MB/s), 114MiB/s-114MiB/s (120MB/s-120MB/s), io=802MiB (840MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751509,
"timestamp_ms" : 1666751509970,
"time" : "Wed Oct 26 04:31:49 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 906781696,
"io_kbytes" : 885529,
"bw_bytes" : 129521739,
"bw" : 126486,
"iops" : 252972.003999,
"runtime" : 7001,
"total_ios" : 1771057,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 210,
"max" : 10360,
"mean" : 266.127212,
"stddev" : 24.148686,
"N" : 1771057
},
"clat_ns" : {
"min" : 281,
"max" : 77735,
"mean" : 7425.552159,
"stddev" : 1737.669617,
"N" : 1771058,
"percentile" : {
"1.000000" : 6816,
"5.000000" : 7008,
"10.000000" : 7008,
"20.000000" : 7072,
"30.000000" : 7072,
"40.000000" : 7136,
"50.000000" : 7200,
"60.000000" : 7200,
"70.000000" : 7264,
"80.000000" : 7328,
"90.000000" : 7392,
"95.000000" : 7520,
"99.000000" : 16320,
"99.500000" : 20864,
"99.900000" : 22144,
"99.950000" : 22912,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 5530,
"max" : 77996,
"mean" : 7691.679367,
"stddev" : 1738.017364,
"N" : 1771058
},
"bw_min" : 126151,
"bw_max" : 126861,
"bw_agg" : 100.000000,
"bw_mean" : 126549.785714,
"bw_dev" : 246.186039,
"bw_samples" : 14,
"iops_min" : 252302,
"iops_max" : 253722,
"iops_mean" : 253099.642857,
"iops_stddev" : 492.334097,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.990401,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.000831,
"20" : 2.473890,
"50" : 0.499589,
"100" : 0.025070,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140424: Wed Oct 26 04:31:49 2022
read: IOPS=253k, BW=124MiB/s (130MB/s)(865MiB/7001msec)
slat (nsec): min=210, max=10360, avg=266.13, stdev=24.15
clat (nsec): min=281, max=77735, avg=7425.55, stdev=1737.67
lat (nsec): min=5530, max=77996, avg=7691.68, stdev=1738.02
clat percentiles (nsec):
| 1.00th=[ 6816], 5.00th=[ 7008], 10.00th=[ 7008], 20.00th=[ 7072],
| 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
| 70.00th=[ 7264], 80.00th=[ 7328], 90.00th=[ 7392], 95.00th=[ 7520],
| 99.00th=[16320], 99.50th=[20864], 99.90th=[22144], 99.95th=[22912],
| 99.99th=[66048]
bw ( KiB/s): min=126151, max=126861, per=100.00%, avg=126549.79, stdev=246.19, samples=14
iops : min=252302, max=253722, avg=253099.64, stdev=492.33, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=97.00%, 20=2.47%, 50=0.50%, 100=0.03%
cpu : usr=99.99%, sys=0.00%, ctx=23, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1771057,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=865MiB (907MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751525,
"timestamp_ms" : 1666751525027,
"time" : "Wed Oct 26 04:32:05 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 834501120,
"io_kbytes" : 814942,
"bw_bytes" : 119197417,
"bw" : 116403,
"iops" : 232807.313241,
"runtime" : 7001,
"total_ios" : 1629884,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 230,
"max" : 27211,
"mean" : 278.884964,
"stddev" : 29.538508,
"N" : 1629884
},
"clat_ns" : {
"min" : 872,
"max" : 121678,
"mean" : 8094.376448,
"stddev" : 2042.240059,
"N" : 1629885,
"percentile" : {
"1.000000" : 6368,
"5.000000" : 6432,
"10.000000" : 6560,
"20.000000" : 7264,
"30.000000" : 7392,
"40.000000" : 7520,
"50.000000" : 7648,
"60.000000" : 7904,
"70.000000" : 8768,
"80.000000" : 8896,
"90.000000" : 8896,
"95.000000" : 9024,
"99.000000" : 18816,
"99.500000" : 20096,
"99.900000" : 22400,
"99.950000" : 28544,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 5881,
"max" : 121959,
"mean" : 8373.261407,
"stddev" : 2043.377005,
"N" : 1629885
},
"bw_min" : 116102,
"bw_max" : 116981,
"bw_agg" : 100.000000,
"bw_mean" : 116429.500000,
"bw_dev" : 245.284372,
"bw_samples" : 14,
"iops_min" : 232204,
"iops_max" : 233962,
"iops_mean" : 232859.357143,
"iops_stddev" : 490.490744,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 99.990551,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.857322,
"20" : 2.646017,
"50" : 0.471567,
"100" : 0.024726,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140474: Wed Oct 26 04:32:05 2022
read: IOPS=233k, BW=114MiB/s (119MB/s)(796MiB/7001msec)
slat (nsec): min=230, max=27211, avg=278.88, stdev=29.54
clat (nsec): min=872, max=121678, avg=8094.38, stdev=2042.24
lat (usec): min=5, max=121, avg= 8.37, stdev= 2.04
clat percentiles (nsec):
| 1.00th=[ 6368], 5.00th=[ 6432], 10.00th=[ 6560], 20.00th=[ 7264],
| 30.00th=[ 7392], 40.00th=[ 7520], 50.00th=[ 7648], 60.00th=[ 7904],
| 70.00th=[ 8768], 80.00th=[ 8896], 90.00th=[ 8896], 95.00th=[ 9024],
| 99.00th=[18816], 99.50th=[20096], 99.90th=[22400], 99.95th=[28544],
| 99.99th=[62720]
bw ( KiB/s): min=116102, max=116981, per=100.00%, avg=116429.50, stdev=245.28, samples=14
iops : min=232204, max=233962, avg=232859.36, stdev=490.49, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 10=96.86%, 20=2.65%, 50=0.47%, 100=0.02%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=23, majf=0, minf=0
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1629884,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s), io=796MiB (835MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751538,
"timestamp_ms" : 1666751538710,
"time" : "Wed Oct 26 04:32:18 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 712784384,
"io_kbytes" : 696078,
"bw_bytes" : 101811796,
"bw" : 99425,
"iops" : 198851.021283,
"runtime" : 7001,
"total_ios" : 1392156,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1282,
"max" : 23945,
"mean" : 1414.855360,
"stddev" : 133.152742,
"N" : 1392156
},
"clat_ns" : {
"min" : 80,
"max" : 100759,
"mean" : 8354.827444,
"stddev" : 2236.779986,
"N" : 1392157,
"percentile" : {
"1.000000" : 5536,
"5.000000" : 6240,
"10.000000" : 6304,
"20.000000" : 6368,
"30.000000" : 6432,
"40.000000" : 6496,
"50.000000" : 8256,
"60.000000" : 9792,
"70.000000" : 9920,
"80.000000" : 10048,
"90.000000" : 10048,
"95.000000" : 10176,
"99.000000" : 16512,
"99.500000" : 17024,
"99.900000" : 19072,
"99.950000" : 23168,
"99.990000" : 60160
}
},
"lat_ns" : {
"min" : 6342,
"max" : 102081,
"mean" : 9769.682853,
"stddev" : 2306.249536,
"N" : 1392157
},
"bw_min" : 99158,
"bw_max" : 99923,
"bw_agg" : 100.000000,
"bw_mean" : 99467.928571,
"bw_dev" : 285.062062,
"bw_samples" : 14,
"iops_min" : 198316,
"iops_max" : 199846,
"iops_mean" : 198936.071429,
"iops_stddev" : 570.072932,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.328571,
"sys_cpu" : 51.842857,
"ctx" : 696330,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 81.288448,
"20" : 18.611635,
"50" : 0.069820,
"100" : 0.020687,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2002080,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10829,
"write_ticks" : 0,
"in_queue" : 10829,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=140513: Wed Oct 26 04:32:18 2022
read: IOPS=199k, BW=97.1MiB/s (102MB/s)(680MiB/7001msec)
slat (nsec): min=1282, max=23945, avg=1414.86, stdev=133.15
clat (nsec): min=80, max=100759, avg=8354.83, stdev=2236.78
lat (usec): min=6, max=102, avg= 9.77, stdev= 2.31
clat percentiles (nsec):
| 1.00th=[ 5536], 5.00th=[ 6240], 10.00th=[ 6304], 20.00th=[ 6368],
| 30.00th=[ 6432], 40.00th=[ 6496], 50.00th=[ 8256], 60.00th=[ 9792],
| 70.00th=[ 9920], 80.00th=[10048], 90.00th=[10048], 95.00th=[10176],
| 99.00th=[16512], 99.50th=[17024], 99.90th=[19072], 99.95th=[23168],
| 99.99th=[60160]
bw ( KiB/s): min=99158, max=99923, per=100.00%, avg=99467.93, stdev=285.06, samples=14
iops : min=198316, max=199846, avg=198936.07, stdev=570.07, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=81.29%, 20=18.61%, 50=0.07%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=18.33%, sys=51.84%, ctx=696330, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1392156,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=97.1MiB/s (102MB/s), 97.1MiB/s-97.1MiB/s (102MB/s-102MB/s), io=680MiB (713MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2002080/0, merge=0/0, ticks=10829/0, in_queue=10829, util=99.09%
$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt': No such file or directory
$ rm /tmp/fio-output_512_2_io_uring-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751552,
"timestamp_ms" : 1666751552714,
"time" : "Wed Oct 26 04:32:32 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 713009152,
"io_kbytes" : 696298,
"bw_bytes" : 101843901,
"bw" : 99456,
"iops" : 198913.726610,
"runtime" : 7001,
"total_ios" : 1392595,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1282,
"max" : 49703,
"mean" : 1410.760628,
"stddev" : 136.155381,
"N" : 1392595
},
"clat_ns" : {
"min" : 80,
"max" : 102632,
"mean" : 8358.082344,
"stddev" : 2240.215117,
"N" : 1392596,
"percentile" : {
"1.000000" : 5536,
"5.000000" : 6304,
"10.000000" : 6304,
"20.000000" : 6368,
"30.000000" : 6432,
"40.000000" : 6496,
"50.000000" : 8256,
"60.000000" : 9792,
"70.000000" : 9920,
"80.000000" : 9920,
"90.000000" : 10048,
"95.000000" : 10176,
"99.000000" : 16512,
"99.500000" : 17024,
"99.900000" : 19072,
"99.950000" : 23168,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 5671,
"max" : 103965,
"mean" : 9768.843017,
"stddev" : 2302.931642,
"N" : 1392596
},
"bw_min" : 99307,
"bw_max" : 99675,
"bw_agg" : 100.000000,
"bw_mean" : 99497.357143,
"bw_dev" : 102.195595,
"bw_samples" : 14,
"iops_min" : 198614,
"iops_max" : 199350,
"iops_mean" : 198994.785714,
"iops_stddev" : 204.325599,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.071429,
"sys_cpu" : 51.157143,
"ctx" : 696426,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 83.180681,
"20" : 16.725394,
"50" : 0.061612,
"100" : 0.023625,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2000153,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10851,
"write_ticks" : 0,
"in_queue" : 10851,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=140551: Wed Oct 26 04:32:32 2022
read: IOPS=199k, BW=97.1MiB/s (102MB/s)(680MiB/7001msec)
slat (nsec): min=1282, max=49703, avg=1410.76, stdev=136.16
clat (nsec): min=80, max=102632, avg=8358.08, stdev=2240.22
lat (usec): min=5, max=103, avg= 9.77, stdev= 2.30
clat percentiles (nsec):
| 1.00th=[ 5536], 5.00th=[ 6304], 10.00th=[ 6304], 20.00th=[ 6368],
| 30.00th=[ 6432], 40.00th=[ 6496], 50.00th=[ 8256], 60.00th=[ 9792],
| 70.00th=[ 9920], 80.00th=[ 9920], 90.00th=[10048], 95.00th=[10176],
| 99.00th=[16512], 99.50th=[17024], 99.90th=[19072], 99.95th=[23168],
| 99.99th=[65280]
bw ( KiB/s): min=99307, max=99675, per=100.00%, avg=99497.36, stdev=102.20, samples=14
iops : min=198614, max=199350, avg=198994.79, stdev=204.33, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=83.18%, 20=16.73%, 50=0.06%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=19.07%, sys=51.16%, ctx=696426, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1392595,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=97.1MiB/s (102MB/s), 97.1MiB/s-97.1MiB/s (102MB/s-102MB/s), io=680MiB (713MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2000153/0, merge=0/0, ticks=10851/0, in_queue=10851, util=99.09%
$ rm /tmp/fio-output_512_2_io_uring-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751567,
"timestamp_ms" : 1666751567468,
"time" : "Wed Oct 26 04:32:47 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 704743424,
"io_kbytes" : 688226,
"bw_bytes" : 100663251,
"bw" : 98303,
"iops" : 196607.770319,
"runtime" : 7001,
"total_ios" : 1376451,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1302,
"max" : 115346,
"mean" : 1448.851801,
"stddev" : 170.761925,
"N" : 1376451
},
"clat_ns" : {
"min" : 90,
"max" : 124113,
"mean" : 8432.095208,
"stddev" : 2364.959566,
"N" : 1376452,
"percentile" : {
"1.000000" : 5344,
"5.000000" : 6304,
"10.000000" : 6304,
"20.000000" : 6368,
"30.000000" : 6432,
"40.000000" : 6496,
"50.000000" : 8896,
"60.000000" : 10048,
"70.000000" : 10048,
"80.000000" : 10176,
"90.000000" : 10176,
"95.000000" : 10304,
"99.000000" : 16512,
"99.500000" : 17280,
"99.900000" : 23168,
"99.950000" : 23424,
"99.990000" : 65280
}
},
"lat_ns" : {
"min" : 6342,
"max" : 125696,
"mean" : 9880.947041,
"stddev" : 2442.322795,
"N" : 1376452
},
"bw_min" : 98131,
"bw_max" : 98526,
"bw_agg" : 100.000000,
"bw_mean" : 98347.384615,
"bw_dev" : 105.430497,
"bw_samples" : 13,
"iops_min" : 196262,
"iops_max" : 197052,
"iops_mean" : 196694.846154,
"iops_stddev" : 210.890116,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.671429,
"sys_cpu" : 52.142857,
"ctx" : 689064,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 59.597109,
"20" : 40.243205,
"50" : 0.122925,
"100" : 0.025064,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1979948,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10785,
"write_ticks" : 0,
"in_queue" : 10785,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=140589: Wed Oct 26 04:32:47 2022
read: IOPS=197k, BW=96.0MiB/s (101MB/s)(672MiB/7001msec)
slat (nsec): min=1302, max=115346, avg=1448.85, stdev=170.76
clat (nsec): min=90, max=124113, avg=8432.10, stdev=2364.96
lat (usec): min=6, max=125, avg= 9.88, stdev= 2.44
clat percentiles (nsec):
| 1.00th=[ 5344], 5.00th=[ 6304], 10.00th=[ 6304], 20.00th=[ 6368],
| 30.00th=[ 6432], 40.00th=[ 6496], 50.00th=[ 8896], 60.00th=[10048],
| 70.00th=[10048], 80.00th=[10176], 90.00th=[10176], 95.00th=[10304],
| 99.00th=[16512], 99.50th=[17280], 99.90th=[23168], 99.95th=[23424],
| 99.99th=[65280]
bw ( KiB/s): min=98131, max=98526, per=100.00%, avg=98347.38, stdev=105.43, samples=13
iops : min=196262, max=197052, avg=196694.85, stdev=210.89, samples=13
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=59.60%, 20=40.24%, 50=0.12%
lat (usec) : 100=0.03%, 250=0.01%
cpu : usr=18.67%, sys=52.14%, ctx=689064, majf=0, minf=1
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1376451,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=96.0MiB/s (101MB/s), 96.0MiB/s-96.0MiB/s (101MB/s-101MB/s), io=672MiB (705MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1979948/0, merge=0/0, ticks=10785/0, in_queue=10785, util=99.09%
$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751582,
"timestamp_ms" : 1666751582489,
"time" : "Wed Oct 26 04:33:02 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 814672896,
"io_kbytes" : 795579,
"bw_bytes" : 116365218,
"bw" : 113637,
"iops" : 227275.674904,
"runtime" : 7001,
"total_ios" : 1591157,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 17412,
"mean" : 151.050024,
"stddev" : 62.073951,
"N" : 1591157
},
"clat_ns" : {
"min" : 5881,
"max" : 120396,
"mean" : 8186.654414,
"stddev" : 1593.597298,
"N" : 1591157,
"percentile" : {
"1.000000" : 7328,
"5.000000" : 7648,
"10.000000" : 7712,
"20.000000" : 7776,
"30.000000" : 7840,
"40.000000" : 7904,
"50.000000" : 7968,
"60.000000" : 8032,
"70.000000" : 8096,
"80.000000" : 8160,
"90.000000" : 8256,
"95.000000" : 8384,
"99.000000" : 16512,
"99.500000" : 19840,
"99.900000" : 20608,
"99.950000" : 21120,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 6071,
"max" : 120566,
"mean" : 8337.704438,
"stddev" : 1596.070054,
"N" : 1591157
},
"bw_min" : 113483,
"bw_max" : 113870,
"bw_agg" : 100.000000,
"bw_mean" : 113676.000000,
"bw_dev" : 107.724006,
"bw_samples" : 14,
"iops_min" : 226966,
"iops_max" : 227740,
"iops_mean" : 227352.071429,
"iops_stddev" : 215.540274,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 33.528571,
"sys_cpu" : 66.442857,
"ctx" : 20,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.203818,
"2" : 50.796182,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.850279,
"20" : 2.729398,
"50" : 0.397949,
"100" : 0.022122,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=140637: Wed Oct 26 04:33:02 2022
read: IOPS=227k, BW=111MiB/s (116MB/s)(777MiB/7001msec)
slat (nsec): min=110, max=17412, avg=151.05, stdev=62.07
clat (usec): min=5, max=120, avg= 8.19, stdev= 1.59
lat (usec): min=6, max=120, avg= 8.34, stdev= 1.60
clat percentiles (nsec):
| 1.00th=[ 7328], 5.00th=[ 7648], 10.00th=[ 7712], 20.00th=[ 7776],
| 30.00th=[ 7840], 40.00th=[ 7904], 50.00th=[ 7968], 60.00th=[ 8032],
| 70.00th=[ 8096], 80.00th=[ 8160], 90.00th=[ 8256], 95.00th=[ 8384],
| 99.00th=[16512], 99.50th=[19840], 99.90th=[20608], 99.95th=[21120],
| 99.99th=[62720]
bw ( KiB/s): min=113483, max=113870, per=100.00%, avg=113676.00, stdev=107.72, samples=14
iops : min=226966, max=227740, avg=227352.07, stdev=215.54, samples=14
lat (usec) : 10=96.85%, 20=2.73%, 50=0.40%, 100=0.02%, 250=0.01%
cpu : usr=33.53%, sys=66.44%, ctx=20, majf=0, minf=0
IO depths : 1=49.2%, 2=50.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1591157,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=111MiB/s (116MB/s), 111MiB/s-111MiB/s (116MB/s-116MB/s), io=777MiB (815MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751597,
"timestamp_ms" : 1666751597582,
"time" : "Wed Oct 26 04:33:17 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 821842432,
"io_kbytes" : 802580,
"bw_bytes" : 117389291,
"bw" : 114637,
"iops" : 229275.817740,
"runtime" : 7001,
"total_ios" : 1605160,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 17734,
"mean" : 149.779642,
"stddev" : 53.132110,
"N" : 1605160
},
"clat_ns" : {
"min" : 5751,
"max" : 97513,
"mean" : 8131.512809,
"stddev" : 1581.103388,
"N" : 1605160,
"percentile" : {
"1.000000" : 7264,
"5.000000" : 7584,
"10.000000" : 7648,
"20.000000" : 7712,
"30.000000" : 7776,
"40.000000" : 7840,
"50.000000" : 7904,
"60.000000" : 7968,
"70.000000" : 8032,
"80.000000" : 8096,
"90.000000" : 8256,
"95.000000" : 8384,
"99.000000" : 16512,
"99.500000" : 19840,
"99.900000" : 20864,
"99.950000" : 21120,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 5952,
"max" : 97693,
"mean" : 8281.292451,
"stddev" : 1583.644423,
"N" : 1605160
},
"bw_min" : 114397,
"bw_max" : 114917,
"bw_agg" : 100.000000,
"bw_mean" : 114668.571429,
"bw_dev" : 150.823745,
"bw_samples" : 14,
"iops_min" : 228794,
"iops_max" : 229834,
"iops_mean" : 229337.214286,
"iops_stddev" : 301.690666,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 35.814286,
"sys_cpu" : 64.157143,
"ctx" : 21,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.234718,
"2" : 50.765282,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.891587,
"20" : 2.622293,
"50" : 0.465873,
"100" : 0.020247,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=140686: Wed Oct 26 04:33:17 2022
read: IOPS=229k, BW=112MiB/s (117MB/s)(784MiB/7001msec)
slat (nsec): min=110, max=17734, avg=149.78, stdev=53.13
clat (nsec): min=5751, max=97513, avg=8131.51, stdev=1581.10
lat (nsec): min=5952, max=97693, avg=8281.29, stdev=1583.64
clat percentiles (nsec):
| 1.00th=[ 7264], 5.00th=[ 7584], 10.00th=[ 7648], 20.00th=[ 7712],
| 30.00th=[ 7776], 40.00th=[ 7840], 50.00th=[ 7904], 60.00th=[ 7968],
| 70.00th=[ 8032], 80.00th=[ 8096], 90.00th=[ 8256], 95.00th=[ 8384],
| 99.00th=[16512], 99.50th=[19840], 99.90th=[20864], 99.95th=[21120],
| 99.99th=[62720]
bw ( KiB/s): min=114397, max=114917, per=100.00%, avg=114668.57, stdev=150.82, samples=14
iops : min=228794, max=229834, avg=229337.21, stdev=301.69, samples=14
lat (usec) : 10=96.89%, 20=2.62%, 50=0.47%, 100=0.02%
cpu : usr=35.81%, sys=64.16%, ctx=21, majf=0, minf=0
IO depths : 1=49.2%, 2=50.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1605160,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=112MiB/s (117MB/s), 112MiB/s-112MiB/s (117MB/s-117MB/s), io=784MiB (822MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751612,
"timestamp_ms" : 1666751612671,
"time" : "Wed Oct 26 04:33:32 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 816369664,
"io_kbytes" : 797236,
"bw_bytes" : 116607579,
"bw" : 113874,
"iops" : 227749.035852,
"runtime" : 7001,
"total_ios" : 1594471,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 23354,
"mean" : 150.289322,
"stddev" : 61.636434,
"N" : 1594471
},
"clat_ns" : {
"min" : 5841,
"max" : 119644,
"mean" : 8172.569133,
"stddev" : 1601.463886,
"N" : 1594471,
"percentile" : {
"1.000000" : 7264,
"5.000000" : 7648,
"10.000000" : 7712,
"20.000000" : 7776,
"30.000000" : 7840,
"40.000000" : 7904,
"50.000000" : 7968,
"60.000000" : 8032,
"70.000000" : 8032,
"80.000000" : 8096,
"90.000000" : 8256,
"95.000000" : 8384,
"99.000000" : 16512,
"99.500000" : 19840,
"99.900000" : 20864,
"99.950000" : 21120,
"99.990000" : 61696
}
},
"lat_ns" : {
"min" : 6092,
"max" : 119814,
"mean" : 8322.858455,
"stddev" : 1603.977324,
"N" : 1594471
},
"bw_min" : 113642,
"bw_max" : 114230,
"bw_agg" : 100.000000,
"bw_mean" : 113906.642857,
"bw_dev" : 147.479650,
"bw_samples" : 14,
"iops_min" : 227284,
"iops_max" : 228460,
"iops_mean" : 227813.357143,
"iops_stddev" : 295.000810,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.100000,
"sys_cpu" : 67.871429,
"ctx" : 27,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 49.172171,
"2" : 50.827829,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.864540,
"20" : 2.690359,
"50" : 0.421707,
"100" : 0.023142,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=140736: Wed Oct 26 04:33:32 2022
read: IOPS=228k, BW=111MiB/s (117MB/s)(779MiB/7001msec)
slat (nsec): min=110, max=23354, avg=150.29, stdev=61.64
clat (usec): min=5, max=119, avg= 8.17, stdev= 1.60
lat (usec): min=6, max=119, avg= 8.32, stdev= 1.60
clat percentiles (nsec):
| 1.00th=[ 7264], 5.00th=[ 7648], 10.00th=[ 7712], 20.00th=[ 7776],
| 30.00th=[ 7840], 40.00th=[ 7904], 50.00th=[ 7968], 60.00th=[ 8032],
| 70.00th=[ 8032], 80.00th=[ 8096], 90.00th=[ 8256], 95.00th=[ 8384],
| 99.00th=[16512], 99.50th=[19840], 99.90th=[20864], 99.95th=[21120],
| 99.99th=[61696]
bw ( KiB/s): min=113642, max=114230, per=100.00%, avg=113906.64, stdev=147.48, samples=14
iops : min=227284, max=228460, avg=227813.36, stdev=295.00, samples=14
lat (usec) : 10=96.86%, 20=2.69%, 50=0.42%, 100=0.02%, 250=0.01%
cpu : usr=32.10%, sys=67.87%, ctx=27, majf=0, minf=0
IO depths : 1=49.2%, 2=50.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1594471,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=111MiB/s (117MB/s), 111MiB/s-111MiB/s (117MB/s-117MB/s), io=779MiB (816MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751627,
"timestamp_ms" : 1666751627772,
"time" : "Wed Oct 26 04:33:47 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 954444288,
"io_kbytes" : 932074,
"bw_bytes" : 136329708,
"bw" : 133134,
"iops" : 266268.675903,
"runtime" : 7001,
"total_ios" : 1864147,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1552,
"max" : 23744,
"mean" : 2227.662047,
"stddev" : 687.257728,
"N" : 1864148
},
"clat_ns" : {
"min" : 241,
"max" : 115497,
"mean" : 5044.965217,
"stddev" : 1900.368884,
"N" : 1864148,
"percentile" : {
"1.000000" : 4048,
"5.000000" : 4080,
"10.000000" : 4128,
"20.000000" : 4128,
"30.000000" : 4192,
"40.000000" : 4256,
"50.000000" : 5216,
"60.000000" : 5280,
"70.000000" : 5344,
"80.000000" : 5408,
"90.000000" : 5472,
"95.000000" : 5664,
"99.000000" : 14272,
"99.500000" : 18304,
"99.900000" : 22400,
"99.950000" : 22912,
"99.990000" : 60672
}
},
"lat_ns" : {
"min" : 5591,
"max" : 117120,
"mean" : 7272.627264,
"stddev" : 1826.947438,
"N" : 1864148
},
"bw_min" : 133014,
"bw_max" : 133288,
"bw_agg" : 100.000000,
"bw_mean" : 133165.500000,
"bw_dev" : 95.984574,
"bw_samples" : 14,
"iops_min" : 266028,
"iops_max" : 266576,
"iops_mean" : 266331.000000,
"iops_stddev" : 191.778318,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 33.700000,
"sys_cpu" : 66.271429,
"ctx" : 25,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.022799,
"10" : 97.619233,
"20" : 1.967710,
"50" : 0.368265,
"100" : 0.019741,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140785: Wed Oct 26 04:33:47 2022
read: IOPS=266k, BW=130MiB/s (136MB/s)(910MiB/7001msec)
slat (nsec): min=1552, max=23744, avg=2227.66, stdev=687.26
clat (nsec): min=241, max=115497, avg=5044.97, stdev=1900.37
lat (usec): min=5, max=117, avg= 7.27, stdev= 1.83
clat percentiles (nsec):
| 1.00th=[ 4048], 5.00th=[ 4080], 10.00th=[ 4128], 20.00th=[ 4128],
| 30.00th=[ 4192], 40.00th=[ 4256], 50.00th=[ 5216], 60.00th=[ 5280],
| 70.00th=[ 5344], 80.00th=[ 5408], 90.00th=[ 5472], 95.00th=[ 5664],
| 99.00th=[14272], 99.50th=[18304], 99.90th=[22400], 99.95th=[22912],
| 99.99th=[60672]
bw ( KiB/s): min=133014, max=133288, per=100.00%, avg=133165.50, stdev=95.98, samples=14
iops : min=266028, max=266576, avg=266331.00, stdev=191.78, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.02%, 10=97.62%, 20=1.97%, 50=0.37%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=33.70%, sys=66.27%, ctx=25, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1864147,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=910MiB (954MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666750994,
"timestamp_ms" : 1666750994677,
"time" : "Wed Oct 26 04:23:14 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 285780992,
"io_kbytes" : 279083,
"bw_bytes" : 40820024,
"bw" : 39863,
"iops" : 79726.467647,
"runtime" : 7001,
"total_ios" : 558165,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2425,
"max" : 28102,
"mean" : 2654.688612,
"stddev" : 235.256413,
"N" : 558166
},
"clat_ns" : {
"min" : 271,
"max" : 80301,
"mean" : 9230.039091,
"stddev" : 1260.026978,
"N" : 558165,
"percentile" : {
"1.000000" : 8896,
"5.000000" : 8896,
"10.000000" : 8896,
"20.000000" : 9024,
"30.000000" : 9024,
"40.000000" : 9024,
"50.000000" : 9024,
"60.000000" : 9024,
"70.000000" : 9152,
"80.000000" : 9152,
"90.000000" : 9152,
"95.000000" : 9408,
"99.000000" : 15296,
"99.500000" : 16064,
"99.900000" : 19328,
"99.950000" : 27520,
"99.990000" : 55552
}
},
"lat_ns" : {
"min" : 9147,
"max" : 82936,
"mean" : 11884.727846,
"stddev" : 1280.549295,
"N" : 558165
},
"bw_min" : 39774,
"bw_max" : 40040,
"bw_agg" : 100.000000,
"bw_mean" : 39873.642857,
"bw_dev" : 66.964522,
"bw_samples" : 14,
"iops_min" : 79548,
"iops_max" : 80080,
"iops_mean" : 79747.285714,
"iops_stddev" : 133.929045,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.842857,
"sys_cpu" : 45.942857,
"ctx" : 558046,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.336988,
"20" : 2.566087,
"50" : 0.071663,
"100" : 0.015587,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=138828: Wed Oct 26 04:23:14 2022
read: IOPS=79.7k, BW=38.9MiB/s (40.8MB/s)(273MiB/7001msec)
slat (nsec): min=2425, max=28102, avg=2654.69, stdev=235.26
clat (nsec): min=271, max=80301, avg=9230.04, stdev=1260.03
lat (nsec): min=9147, max=82936, avg=11884.73, stdev=1280.55
clat percentiles (nsec):
| 1.00th=[ 8896], 5.00th=[ 8896], 10.00th=[ 8896], 20.00th=[ 9024],
| 30.00th=[ 9024], 40.00th=[ 9024], 50.00th=[ 9024], 60.00th=[ 9024],
| 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9152], 95.00th=[ 9408],
| 99.00th=[15296], 99.50th=[16064], 99.90th=[19328], 99.95th=[27520],
| 99.99th=[55552]
bw ( KiB/s): min=39774, max=40040, per=100.00%, avg=39873.64, stdev=66.96, samples=14
iops : min=79548, max=80080, avg=79747.29, stdev=133.93, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=97.34%, 20=2.57%, 50=0.07%, 100=0.02%
cpu : usr=13.84%, sys=45.94%, ctx=558046, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=558165,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=38.9MiB/s (40.8MB/s), 38.9MiB/s-38.9MiB/s (40.8MB/s-40.8MB/s), io=273MiB (286MB), run=7001-7001msec
$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751642,
"timestamp_ms" : 1666751642852,
"time" : "Wed Oct 26 04:34:02 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 955027456,
"io_kbytes" : 932644,
"bw_bytes" : 136413006,
"bw" : 133215,
"iops" : 266431.366948,
"runtime" : 7001,
"total_ios" : 1865286,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1533,
"max" : 25778,
"mean" : 2232.692276,
"stddev" : 689.704731,
"N" : 1865287
},
"clat_ns" : {
"min" : 231,
"max" : 102753,
"mean" : 5046.237951,
"stddev" : 1921.027764,
"N" : 1865287,
"percentile" : {
"1.000000" : 4048,
"5.000000" : 4080,
"10.000000" : 4128,
"20.000000" : 4128,
"30.000000" : 4192,
"40.000000" : 4256,
"50.000000" : 5216,
"60.000000" : 5280,
"70.000000" : 5344,
"80.000000" : 5408,
"90.000000" : 5472,
"95.000000" : 5664,
"99.000000" : 14400,
"99.500000" : 18304,
"99.900000" : 22656,
"99.950000" : 22912,
"99.990000" : 61696
}
},
"lat_ns" : {
"min" : 5651,
"max" : 104316,
"mean" : 7278.930227,
"stddev" : 1850.805954,
"N" : 1865287
},
"bw_min" : 132892,
"bw_max" : 133392,
"bw_agg" : 100.000000,
"bw_mean" : 133246.500000,
"bw_dev" : 157.042253,
"bw_samples" : 14,
"iops_min" : 265784,
"iops_max" : 266785,
"iops_mean" : 266493.214286,
"iops_stddev" : 314.111734,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 35.300000,
"sys_cpu" : 64.671429,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.024286,
"10" : 97.589217,
"20" : 1.995404,
"50" : 0.367129,
"100" : 0.021820,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140835: Wed Oct 26 04:34:02 2022
read: IOPS=266k, BW=130MiB/s (136MB/s)(911MiB/7001msec)
slat (nsec): min=1533, max=25778, avg=2232.69, stdev=689.70
clat (nsec): min=231, max=102753, avg=5046.24, stdev=1921.03
lat (usec): min=5, max=104, avg= 7.28, stdev= 1.85
clat percentiles (nsec):
| 1.00th=[ 4048], 5.00th=[ 4080], 10.00th=[ 4128], 20.00th=[ 4128],
| 30.00th=[ 4192], 40.00th=[ 4256], 50.00th=[ 5216], 60.00th=[ 5280],
| 70.00th=[ 5344], 80.00th=[ 5408], 90.00th=[ 5472], 95.00th=[ 5664],
| 99.00th=[14400], 99.50th=[18304], 99.90th=[22656], 99.95th=[22912],
| 99.99th=[61696]
bw ( KiB/s): min=132892, max=133392, per=100.00%, avg=133246.50, stdev=157.04, samples=14
iops : min=265784, max=266785, avg=266493.21, stdev=314.11, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.02%, 10=97.59%, 20=2.00%, 50=0.37%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=35.30%, sys=64.67%, ctx=23, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1865286,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=911MiB (955MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751657,
"timestamp_ms" : 1666751657907,
"time" : "Wed Oct 26 04:34:17 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 949441536,
"io_kbytes" : 927189,
"bw_bytes" : 135615131,
"bw" : 132436,
"iops" : 264873.018140,
"runtime" : 7001,
"total_ios" : 1854376,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1552,
"max" : 24926,
"mean" : 2245.204077,
"stddev" : 687.051828,
"N" : 1854377
},
"clat_ns" : {
"min" : 230,
"max" : 103073,
"mean" : 5066.601525,
"stddev" : 1903.677678,
"N" : 1854377,
"percentile" : {
"1.000000" : 4080,
"5.000000" : 4128,
"10.000000" : 4128,
"20.000000" : 4192,
"30.000000" : 4192,
"40.000000" : 4256,
"50.000000" : 5216,
"60.000000" : 5280,
"70.000000" : 5408,
"80.000000" : 5472,
"90.000000" : 5536,
"95.000000" : 5664,
"99.000000" : 14400,
"99.500000" : 18304,
"99.900000" : 22400,
"99.950000" : 22912,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5641,
"max" : 104686,
"mean" : 7311.805603,
"stddev" : 1830.883007,
"N" : 1854377
},
"bw_min" : 132278,
"bw_max" : 133014,
"bw_agg" : 100.000000,
"bw_mean" : 132478.384615,
"bw_dev" : 208.302880,
"bw_samples" : 13,
"iops_min" : 264556,
"iops_max" : 266028,
"iops_mean" : 264957.000000,
"iops_stddev" : 416.682533,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.785714,
"sys_cpu" : 69.185714,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.017310,
"10" : 97.592667,
"20" : 1.983039,
"50" : 0.384172,
"100" : 0.020600,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140884: Wed Oct 26 04:34:17 2022
read: IOPS=265k, BW=129MiB/s (136MB/s)(905MiB/7001msec)
slat (nsec): min=1552, max=24926, avg=2245.20, stdev=687.05
clat (nsec): min=230, max=103073, avg=5066.60, stdev=1903.68
lat (usec): min=5, max=104, avg= 7.31, stdev= 1.83
clat percentiles (nsec):
| 1.00th=[ 4080], 5.00th=[ 4128], 10.00th=[ 4128], 20.00th=[ 4192],
| 30.00th=[ 4192], 40.00th=[ 4256], 50.00th=[ 5216], 60.00th=[ 5280],
| 70.00th=[ 5408], 80.00th=[ 5472], 90.00th=[ 5536], 95.00th=[ 5664],
| 99.00th=[14400], 99.50th=[18304], 99.90th=[22400], 99.95th=[22912],
| 99.99th=[61184]
bw ( KiB/s): min=132278, max=133014, per=100.00%, avg=132478.38, stdev=208.30, samples=13
iops : min=264556, max=266028, avg=264957.00, stdev=416.68, samples=13
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.02%, 10=97.59%, 20=1.98%, 50=0.38%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=30.79%, sys=69.19%, ctx=23, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1854376,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=129MiB/s (136MB/s), 129MiB/s-129MiB/s (136MB/s-136MB/s), io=905MiB (949MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751671,
"timestamp_ms" : 1666751671714,
"time" : "Wed Oct 26 04:34:31 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 764834304,
"io_kbytes" : 746908,
"bw_bytes" : 109246436,
"bw" : 106685,
"iops" : 213371.804028,
"runtime" : 7001,
"total_ios" : 1493816,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1382,
"max" : 25729,
"mean" : 1532.744973,
"stddev" : 144.972706,
"N" : 1493816
},
"clat_ns" : {
"min" : 130,
"max" : 117480,
"mean" : 7572.991351,
"stddev" : 2518.779534,
"N" : 1493817,
"percentile" : {
"1.000000" : 5152,
"5.000000" : 5152,
"10.000000" : 5216,
"20.000000" : 5280,
"30.000000" : 5344,
"40.000000" : 5472,
"50.000000" : 9024,
"60.000000" : 9024,
"70.000000" : 9152,
"80.000000" : 9280,
"90.000000" : 9408,
"95.000000" : 10432,
"99.000000" : 17024,
"99.500000" : 18048,
"99.900000" : 19840,
"99.950000" : 23168,
"99.990000" : 60160
}
},
"lat_ns" : {
"min" : 6092,
"max" : 119193,
"mean" : 9105.736391,
"stddev" : 2595.087936,
"N" : 1493817
},
"bw_min" : 106453,
"bw_max" : 106968,
"bw_agg" : 100.000000,
"bw_mean" : 106711.785714,
"bw_dev" : 136.440000,
"bw_samples" : 14,
"iops_min" : 212906,
"iops_max" : 213936,
"iops_mean" : 213423.428571,
"iops_stddev" : 272.960328,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.857143,
"sys_cpu" : 57.628571,
"ctx" : 747066,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 94.517866,
"20" : 5.375160,
"50" : 0.077787,
"100" : 0.020351,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=140924: Wed Oct 26 04:34:31 2022
read: IOPS=213k, BW=104MiB/s (109MB/s)(729MiB/7001msec)
slat (nsec): min=1382, max=25729, avg=1532.74, stdev=144.97
clat (nsec): min=130, max=117480, avg=7572.99, stdev=2518.78
lat (usec): min=6, max=119, avg= 9.11, stdev= 2.60
clat percentiles (nsec):
| 1.00th=[ 5152], 5.00th=[ 5152], 10.00th=[ 5216], 20.00th=[ 5280],
| 30.00th=[ 5344], 40.00th=[ 5472], 50.00th=[ 9024], 60.00th=[ 9024],
| 70.00th=[ 9152], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[10432],
| 99.00th=[17024], 99.50th=[18048], 99.90th=[19840], 99.95th=[23168],
| 99.99th=[60160]
bw ( KiB/s): min=106453, max=106968, per=100.00%, avg=106711.79, stdev=136.44, samples=14
iops : min=212906, max=213936, avg=213423.43, stdev=272.96, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=94.52%, 20=5.38%, 50=0.08%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=19.86%, sys=57.63%, ctx=747066, majf=0, minf=1
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1493816,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=729MiB (765MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_io_uring-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751685,
"timestamp_ms" : 1666751685358,
"time" : "Wed Oct 26 04:34:45 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 765723648,
"io_kbytes" : 747777,
"bw_bytes" : 109373467,
"bw" : 106810,
"iops" : 213619.911441,
"runtime" : 7001,
"total_ios" : 1495553,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1352,
"max" : 22452,
"mean" : 1493.274781,
"stddev" : 148.492491,
"N" : 1495553
},
"clat_ns" : {
"min" : 130,
"max" : 122009,
"mean" : 7620.069698,
"stddev" : 2548.142265,
"N" : 1495554,
"percentile" : {
"1.000000" : 5152,
"5.000000" : 5216,
"10.000000" : 5216,
"20.000000" : 5344,
"30.000000" : 5408,
"40.000000" : 5536,
"50.000000" : 8896,
"60.000000" : 9024,
"70.000000" : 9024,
"80.000000" : 9152,
"90.000000" : 9408,
"95.000000" : 10560,
"99.000000" : 17280,
"99.500000" : 18048,
"99.900000" : 20608,
"99.950000" : 23168,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 6161,
"max" : 123401,
"mean" : 9113.344539,
"stddev" : 2616.635881,
"N" : 1495554
},
"bw_min" : 106727,
"bw_max" : 106938,
"bw_agg" : 100.000000,
"bw_mean" : 106835.214286,
"bw_dev" : 72.031167,
"bw_samples" : 14,
"iops_min" : 213454,
"iops_max" : 213876,
"iops_mean" : 213670.500000,
"iops_stddev" : 144.097590,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.214286,
"sys_cpu" : 60.571429,
"ctx" : 748054,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 91.554763,
"20" : 8.329695,
"50" : 0.084517,
"100" : 0.016984,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=140964: Wed Oct 26 04:34:45 2022
read: IOPS=214k, BW=104MiB/s (109MB/s)(730MiB/7001msec)
slat (nsec): min=1352, max=22452, avg=1493.27, stdev=148.49
clat (nsec): min=130, max=122009, avg=7620.07, stdev=2548.14
lat (usec): min=6, max=123, avg= 9.11, stdev= 2.62
clat percentiles (nsec):
| 1.00th=[ 5152], 5.00th=[ 5216], 10.00th=[ 5216], 20.00th=[ 5344],
| 30.00th=[ 5408], 40.00th=[ 5536], 50.00th=[ 8896], 60.00th=[ 9024],
| 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9408], 95.00th=[10560],
| 99.00th=[17280], 99.50th=[18048], 99.90th=[20608], 99.95th=[23168],
| 99.99th=[61184]
bw ( KiB/s): min=106727, max=106938, per=100.00%, avg=106835.21, stdev=72.03, samples=14
iops : min=213454, max=213876, avg=213670.50, stdev=144.10, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=91.55%, 20=8.33%, 50=0.08%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=16.21%, sys=60.57%, ctx=748054, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1495553,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=730MiB (766MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt': No such file or directory
$ rm /tmp/fio-output_512_2_io_uring-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751698,
"timestamp_ms" : 1666751698998,
"time" : "Wed Oct 26 04:34:58 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 762477568,
"io_kbytes" : 744607,
"bw_bytes" : 108909808,
"bw" : 106357,
"iops" : 212714.326525,
"runtime" : 7001,
"total_ios" : 1489213,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1352,
"max" : 26169,
"mean" : 1505.332639,
"stddev" : 162.198529,
"N" : 1489213
},
"clat_ns" : {
"min" : 140,
"max" : 121859,
"mean" : 7639.105499,
"stddev" : 2552.195648,
"N" : 1489214,
"percentile" : {
"1.000000" : 5088,
"5.000000" : 5152,
"10.000000" : 5216,
"20.000000" : 5280,
"30.000000" : 5344,
"40.000000" : 5600,
"50.000000" : 8896,
"60.000000" : 9024,
"70.000000" : 9024,
"80.000000" : 9152,
"90.000000" : 9664,
"95.000000" : 10560,
"99.000000" : 17280,
"99.500000" : 18048,
"99.900000" : 20608,
"99.950000" : 23168,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 6061,
"max" : 123251,
"mean" : 9144.438204,
"stddev" : 2629.067758,
"N" : 1489214
},
"bw_min" : 105852,
"bw_max" : 106636,
"bw_agg" : 100.000000,
"bw_mean" : 106381.928571,
"bw_dev" : 196.151615,
"bw_samples" : 14,
"iops_min" : 211705,
"iops_max" : 213272,
"iops_mean" : 212763.928571,
"iops_stddev" : 392.095448,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.128571,
"sys_cpu" : 59.557143,
"ctx" : 745014,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 90.268484,
"20" : 9.615884,
"50" : 0.085481,
"100" : 0.017727,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=141004: Wed Oct 26 04:34:58 2022
read: IOPS=213k, BW=104MiB/s (109MB/s)(727MiB/7001msec)
slat (nsec): min=1352, max=26169, avg=1505.33, stdev=162.20
clat (nsec): min=140, max=121859, avg=7639.11, stdev=2552.20
lat (usec): min=6, max=123, avg= 9.14, stdev= 2.63
clat percentiles (nsec):
| 1.00th=[ 5088], 5.00th=[ 5152], 10.00th=[ 5216], 20.00th=[ 5280],
| 30.00th=[ 5344], 40.00th=[ 5600], 50.00th=[ 8896], 60.00th=[ 9024],
| 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9664], 95.00th=[10560],
| 99.00th=[17280], 99.50th=[18048], 99.90th=[20608], 99.95th=[23168],
| 99.99th=[62720]
bw ( KiB/s): min=105852, max=106636, per=100.00%, avg=106381.93, stdev=196.15, samples=14
iops : min=211705, max=213272, avg=212763.93, stdev=392.10, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 4=0.01%, 10=90.27%, 20=9.62%, 50=0.09%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=17.13%, sys=59.56%, ctx=745014, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1489213,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=727MiB (762MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-reference_0.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751712,
"timestamp_ms" : 1666751712628,
"time" : "Wed Oct 26 04:35:12 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 779892224,
"io_kbytes" : 761613,
"bw_bytes" : 111397260,
"bw" : 108786,
"iops" : 217572.489644,
"runtime" : 7001,
"total_ios" : 1523225,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1392,
"max" : 26299,
"mean" : 1533.941948,
"stddev" : 145.787725,
"N" : 1523226
},
"clat_ns" : {
"min" : 401,
"max" : 140834,
"mean" : 7415.475688,
"stddev" : 1799.292095,
"N" : 1523226,
"percentile" : {
"1.000000" : 6880,
"5.000000" : 6944,
"10.000000" : 6944,
"20.000000" : 7008,
"30.000000" : 7072,
"40.000000" : 7072,
"50.000000" : 7136,
"60.000000" : 7200,
"70.000000" : 7264,
"80.000000" : 7328,
"90.000000" : 7392,
"95.000000" : 7648,
"99.000000" : 15808,
"99.500000" : 19328,
"99.900000" : 21120,
"99.950000" : 25472,
"99.990000" : 64768
}
},
"lat_ns" : {
"min" : 5911,
"max" : 142467,
"mean" : 8949.417635,
"stddev" : 1803.451576,
"N" : 1523226
},
"bw_min" : 108527,
"bw_max" : 109183,
"bw_agg" : 100.000000,
"bw_mean" : 108831.785714,
"bw_dev" : 172.703471,
"bw_samples" : 14,
"iops_min" : 217054,
"iops_max" : 218366,
"iops_mean" : 217663.714286,
"iops_stddev" : 345.357881,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 23.371429,
"sys_cpu" : 54.428571,
"ctx" : 761236,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 97.033400,
"20" : 2.567119,
"50" : 0.367510,
"100" : 0.019761,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2179709,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10715,
"write_ticks" : 0,
"in_queue" : 10715,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=141045: Wed Oct 26 04:35:12 2022
read: IOPS=218k, BW=106MiB/s (111MB/s)(744MiB/7001msec)
slat (nsec): min=1392, max=26299, avg=1533.94, stdev=145.79
clat (nsec): min=401, max=140834, avg=7415.48, stdev=1799.29
lat (usec): min=5, max=142, avg= 8.95, stdev= 1.80
clat percentiles (nsec):
| 1.00th=[ 6880], 5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
| 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
| 70.00th=[ 7264], 80.00th=[ 7328], 90.00th=[ 7392], 95.00th=[ 7648],
| 99.00th=[15808], 99.50th=[19328], 99.90th=[21120], 99.95th=[25472],
| 99.99th=[64768]
bw ( KiB/s): min=108527, max=109183, per=100.00%, avg=108831.79, stdev=172.70, samples=14
iops : min=217054, max=218366, avg=217663.71, stdev=345.36, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=97.03%, 20=2.57%, 50=0.37%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=23.37%, sys=54.43%, ctx=761236, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1523225,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=744MiB (780MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2179709/0, merge=0/0, ticks=10715/0, in_queue=10715, util=99.09%
$ rm /tmp/fio-output_512_2_libaio-reference_1.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751726,
"timestamp_ms" : 1666751726267,
"time" : "Wed Oct 26 04:35:26 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 770420224,
"io_kbytes" : 752363,
"bw_bytes" : 110044311,
"bw" : 107465,
"iops" : 214930.009999,
"runtime" : 7001,
"total_ios" : 1504725,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1412,
"max" : 15820,
"mean" : 1557.443289,
"stddev" : 138.168896,
"N" : 1504726
},
"clat_ns" : {
"min" : 400,
"max" : 123452,
"mean" : 7487.520184,
"stddev" : 1736.521261,
"N" : 1504726,
"percentile" : {
"1.000000" : 6944,
"5.000000" : 7008,
"10.000000" : 7008,
"20.000000" : 7072,
"30.000000" : 7136,
"40.000000" : 7200,
"50.000000" : 7264,
"60.000000" : 7264,
"70.000000" : 7328,
"80.000000" : 7392,
"90.000000" : 7456,
"95.000000" : 7712,
"99.000000" : 15680,
"99.500000" : 19328,
"99.900000" : 20608,
"99.950000" : 24448,
"99.990000" : 64256
}
},
"lat_ns" : {
"min" : 5961,
"max" : 125115,
"mean" : 9044.963474,
"stddev" : 1739.023859,
"N" : 1504726
},
"bw_min" : 107304,
"bw_max" : 107838,
"bw_agg" : 100.000000,
"bw_mean" : 107519.384615,
"bw_dev" : 142.703970,
"bw_samples" : 13,
"iops_min" : 214608,
"iops_max" : 215674,
"iops_mean" : 215038.615385,
"iops_stddev" : 285.036120,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 23.071429,
"sys_cpu" : 55.214286,
"ctx" : 751974,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.981841,
"20" : 2.622007,
"50" : 0.365117,
"100" : 0.019671,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2102056,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10370,
"write_ticks" : 0,
"in_queue" : 10370,
"util" : 99.085923
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=141084: Wed Oct 26 04:35:26 2022
read: IOPS=215k, BW=105MiB/s (110MB/s)(735MiB/7001msec)
slat (nsec): min=1412, max=15820, avg=1557.44, stdev=138.17
clat (nsec): min=400, max=123452, avg=7487.52, stdev=1736.52
lat (usec): min=5, max=125, avg= 9.04, stdev= 1.74
clat percentiles (nsec):
| 1.00th=[ 6944], 5.00th=[ 7008], 10.00th=[ 7008], 20.00th=[ 7072],
| 30.00th=[ 7136], 40.00th=[ 7200], 50.00th=[ 7264], 60.00th=[ 7264],
| 70.00th=[ 7328], 80.00th=[ 7392], 90.00th=[ 7456], 95.00th=[ 7712],
| 99.00th=[15680], 99.50th=[19328], 99.90th=[20608], 99.95th=[24448],
| 99.99th=[64256]
bw ( KiB/s): min=107304, max=107838, per=100.00%, avg=107519.38, stdev=142.70, samples=13
iops : min=214608, max=215674, avg=215038.62, stdev=285.04, samples=13
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=96.98%, 20=2.62%, 50=0.37%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=23.07%, sys=55.21%, ctx=751974, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1504725,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=735MiB (770MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2102056/0, merge=0/0, ticks=10370/0, in_queue=10370, util=99.09%
$ rm /tmp/fio-output_512_2_libaio-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751739,
"timestamp_ms" : 1666751739922,
"time" : "Wed Oct 26 04:35:39 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 771123712,
"io_kbytes" : 753050,
"bw_bytes" : 110144795,
"bw" : 107563,
"iops" : 215126.410513,
"runtime" : 7001,
"total_ios" : 1506100,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1392,
"max" : 26369,
"mean" : 1556.186424,
"stddev" : 150.779054,
"N" : 1506100
},
"clat_ns" : {
"min" : 420,
"max" : 149550,
"mean" : 7484.908665,
"stddev" : 1738.332903,
"N" : 1506101,
"percentile" : {
"1.000000" : 6944,
"5.000000" : 6944,
"10.000000" : 7008,
"20.000000" : 7072,
"30.000000" : 7136,
"40.000000" : 7200,
"50.000000" : 7264,
"60.000000" : 7264,
"70.000000" : 7328,
"80.000000" : 7392,
"90.000000" : 7520,
"95.000000" : 7776,
"99.000000" : 15680,
"99.500000" : 18816,
"99.900000" : 20864,
"99.950000" : 24960,
"99.990000" : 62208
}
},
"lat_ns" : {
"min" : 5911,
"max" : 151223,
"mean" : 9041.095007,
"stddev" : 1740.860134,
"N" : 1506101
},
"bw_min" : 107410,
"bw_max" : 107798,
"bw_agg" : 100.000000,
"bw_mean" : 107620.692308,
"bw_dev" : 114.054362,
"bw_samples" : 13,
"iops_min" : 214820,
"iops_max" : 215596,
"iops_mean" : 215241.230769,
"iops_stddev" : 227.980319,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 25.185714,
"sys_cpu" : 52.814286,
"ctx" : 752769,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.987119,
"20" : 2.608924,
"50" : 0.373880,
"100" : 0.017197,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2105479,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 10382,
"write_ticks" : 0,
"in_queue" : 10382,
"util" : 99.085923
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=141122: Wed Oct 26 04:35:39 2022
read: IOPS=215k, BW=105MiB/s (110MB/s)(735MiB/7001msec)
slat (nsec): min=1392, max=26369, avg=1556.19, stdev=150.78
clat (nsec): min=420, max=149550, avg=7484.91, stdev=1738.33
lat (usec): min=5, max=151, avg= 9.04, stdev= 1.74
clat percentiles (nsec):
| 1.00th=[ 6944], 5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7072],
| 30.00th=[ 7136], 40.00th=[ 7200], 50.00th=[ 7264], 60.00th=[ 7264],
| 70.00th=[ 7328], 80.00th=[ 7392], 90.00th=[ 7520], 95.00th=[ 7776],
| 99.00th=[15680], 99.50th=[18816], 99.90th=[20864], 99.95th=[24960],
| 99.99th=[62208]
bw ( KiB/s): min=107410, max=107798, per=100.00%, avg=107620.69, stdev=114.05, samples=13
iops : min=214820, max=215596, avg=215241.23, stdev=227.98, samples=13
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=96.99%, 20=2.61%, 50=0.37%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=25.19%, sys=52.81%, ctx=752769, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1506100,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=735MiB (771MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2105479/0, merge=0/0, ticks=10382/0, in_queue=10382, util=99.09%
$ rm /tmp/fio-output_512_2_libaio-bdev_aio_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751754,
"timestamp_ms" : 1666751754974,
"time" : "Wed Oct 26 04:35:54 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 925439488,
"io_kbytes" : 903749,
"bw_bytes" : 132186757,
"bw" : 129088,
"iops" : 258177.117555,
"runtime" : 7001,
"total_ios" : 1807498,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1603,
"max" : 30868,
"mean" : 2315.003258,
"stddev" : 734.458639,
"N" : 1807498
},
"clat_ns" : {
"min" : 220,
"max" : 153418,
"mean" : 5163.300420,
"stddev" : 1884.570066,
"N" : 1807499,
"percentile" : {
"1.000000" : 4128,
"5.000000" : 4128,
"10.000000" : 4192,
"20.000000" : 4192,
"30.000000" : 4256,
"40.000000" : 4384,
"50.000000" : 5344,
"60.000000" : 5408,
"70.000000" : 5536,
"80.000000" : 5600,
"90.000000" : 5792,
"95.000000" : 5920,
"99.000000" : 13376,
"99.500000" : 18048,
"99.900000" : 20608,
"99.950000" : 20864,
"99.990000" : 63744
}
},
"lat_ns" : {
"min" : 5731,
"max" : 155091,
"mean" : 7478.303306,
"stddev" : 1803.617973,
"N" : 1807499
},
"bw_min" : 128902,
"bw_max" : 129385,
"bw_agg" : 100.000000,
"bw_mean" : 129123.500000,
"bw_dev" : 156.196006,
"bw_samples" : 14,
"iops_min" : 257804,
"iops_max" : 258770,
"iops_mean" : 258247.000000,
"iops_stddev" : 312.392012,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 57.414286,
"sys_cpu" : 42.557143,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 1.656212,
"2" : 98.343788,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.012172,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 97.917453,
"20" : 1.799283,
"50" : 0.244924,
"100" : 0.017317,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=141171: Wed Oct 26 04:35:54 2022
read: IOPS=258k, BW=126MiB/s (132MB/s)(883MiB/7001msec)
slat (nsec): min=1603, max=30868, avg=2315.00, stdev=734.46
clat (nsec): min=220, max=153418, avg=5163.30, stdev=1884.57
lat (usec): min=5, max=155, avg= 7.48, stdev= 1.80
clat percentiles (nsec):
| 1.00th=[ 4128], 5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4192],
| 30.00th=[ 4256], 40.00th=[ 4384], 50.00th=[ 5344], 60.00th=[ 5408],
| 70.00th=[ 5536], 80.00th=[ 5600], 90.00th=[ 5792], 95.00th=[ 5920],
| 99.00th=[13376], 99.50th=[18048], 99.90th=[20608], 99.95th=[20864],
| 99.99th=[63744]
bw ( KiB/s): min=128902, max=129385, per=100.00%, avg=129123.50, stdev=156.20, samples=14
iops : min=257804, max=258770, avg=258247.00, stdev=312.39, samples=14
lat (nsec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=97.92%, 20=1.80%, 50=0.24%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=57.41%, sys=42.56%, ctx=23, majf=0, minf=0
IO depths : 1=1.7%, 2=98.3%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1807498,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=883MiB (925MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_aio_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751770,
"timestamp_ms" : 1666751770080,
"time" : "Wed Oct 26 04:36:10 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 910187008,
"io_kbytes" : 888854,
"bw_bytes" : 130008142,
"bw" : 126961,
"iops" : 253921.868305,
"runtime" : 7001,
"total_ios" : 1777707,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1593,
"max" : 30537,
"mean" : 2352.376895,
"stddev" : 772.406286,
"N" : 1777708
},
"clat_ns" : {
"min" : 241,
"max" : 135454,
"mean" : 5251.168977,
"stddev" : 1853.294464,
"N" : 1777707,
"percentile" : {
"1.000000" : 4192,
"5.000000" : 4192,
"10.000000" : 4256,
"20.000000" : 4256,
"30.000000" : 4320,
"40.000000" : 4448,
"50.000000" : 5472,
"60.000000" : 5536,
"70.000000" : 5664,
"80.000000" : 5728,
"90.000000" : 5920,
"95.000000" : 5984,
"99.000000" : 13376,
"99.500000" : 17792,
"99.900000" : 20608,
"99.950000" : 20864,
"99.990000" : 60160
}
},
"lat_ns" : {
"min" : 5781,
"max" : 137148,
"mean" : 7603.546254,
"stddev" : 1761.177099,
"N" : 1777707
},
"bw_min" : 126711,
"bw_max" : 127293,
"bw_agg" : 100.000000,
"bw_mean" : 126994.142857,
"bw_dev" : 203.450942,
"bw_samples" : 14,
"iops_min" : 253422,
"iops_max" : 254586,
"iops_mean" : 253988.357143,
"iops_stddev" : 407.005879,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 58.271429,
"sys_cpu" : 41.700000,
"ctx" : 20,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 1.653985,
"2" : 98.346015,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.014457,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 97.943981,
"20" : 1.808003,
"50" : 0.211565,
"100" : 0.018394,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=141222: Wed Oct 26 04:36:10 2022
read: IOPS=254k, BW=124MiB/s (130MB/s)(868MiB/7001msec)
slat (nsec): min=1593, max=30537, avg=2352.38, stdev=772.41
clat (nsec): min=241, max=135454, avg=5251.17, stdev=1853.29
lat (usec): min=5, max=137, avg= 7.60, stdev= 1.76
clat percentiles (nsec):
| 1.00th=[ 4192], 5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
| 30.00th=[ 4320], 40.00th=[ 4448], 50.00th=[ 5472], 60.00th=[ 5536],
| 70.00th=[ 5664], 80.00th=[ 5728], 90.00th=[ 5920], 95.00th=[ 5984],
| 99.00th=[13376], 99.50th=[17792], 99.90th=[20608], 99.95th=[20864],
| 99.99th=[60160]
bw ( KiB/s): min=126711, max=127293, per=100.00%, avg=126994.14, stdev=203.45, samples=14
iops : min=253422, max=254586, avg=253988.36, stdev=407.01, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=97.94%, 20=1.81%, 50=0.21%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=58.27%, sys=41.70%, ctx=20, majf=0, minf=0
IO depths : 1=1.7%, 2=98.3%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1777707,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=868MiB (910MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_aio_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751008,
"timestamp_ms" : 1666751008324,
"time" : "Wed Oct 26 04:23:28 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 291316224,
"io_kbytes" : 284488,
"bw_bytes" : 41610659,
"bw" : 40635,
"iops" : 81270.675618,
"runtime" : 7001,
"total_ios" : 568976,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2325,
"max" : 28083,
"mean" : 2529.571946,
"stddev" : 230.071012,
"N" : 568977
},
"clat_ns" : {
"min" : 270,
"max" : 111159,
"mean" : 9106.443340,
"stddev" : 1251.821492,
"N" : 568976,
"percentile" : {
"1.000000" : 8768,
"5.000000" : 8768,
"10.000000" : 8768,
"20.000000" : 8896,
"30.000000" : 8896,
"40.000000" : 8896,
"50.000000" : 8896,
"60.000000" : 8896,
"70.000000" : 9024,
"80.000000" : 9024,
"90.000000" : 9024,
"95.000000" : 9280,
"99.000000" : 15552,
"99.500000" : 16512,
"99.900000" : 19072,
"99.950000" : 21632,
"99.990000" : 55552
}
},
"lat_ns" : {
"min" : 8857,
"max" : 113743,
"mean" : 11636.015347,
"stddev" : 1270.086719,
"N" : 568976
},
"bw_min" : 40559,
"bw_max" : 40758,
"bw_agg" : 100.000000,
"bw_mean" : 40646.000000,
"bw_dev" : 70.886692,
"bw_samples" : 14,
"iops_min" : 81118,
"iops_max" : 81516,
"iops_mean" : 81292.000000,
"iops_stddev" : 141.773384,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.200000,
"sys_cpu" : 43.085714,
"ctx" : 568842,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.012479,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.371067,
"20" : 2.537365,
"50" : 0.064678,
"100" : 0.014236,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=138867: Wed Oct 26 04:23:28 2022
read: IOPS=81.3k, BW=39.7MiB/s (41.6MB/s)(278MiB/7001msec)
slat (nsec): min=2325, max=28083, avg=2529.57, stdev=230.07
clat (nsec): min=270, max=111159, avg=9106.44, stdev=1251.82
lat (usec): min=8, max=113, avg=11.64, stdev= 1.27
clat percentiles (nsec):
| 1.00th=[ 8768], 5.00th=[ 8768], 10.00th=[ 8768], 20.00th=[ 8896],
| 30.00th=[ 8896], 40.00th=[ 8896], 50.00th=[ 8896], 60.00th=[ 8896],
| 70.00th=[ 9024], 80.00th=[ 9024], 90.00th=[ 9024], 95.00th=[ 9280],
| 99.00th=[15552], 99.50th=[16512], 99.90th=[19072], 99.95th=[21632],
| 99.99th=[55552]
bw ( KiB/s): min=40559, max=40758, per=100.00%, avg=40646.00, stdev=70.89, samples=14
iops : min=81118, max=81516, avg=81292.00, stdev=141.77, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=97.37%, 20=2.54%, 50=0.06%, 100=0.01%, 250=0.01%
cpu : usr=16.20%, sys=43.09%, ctx=568842, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=568976,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=39.7MiB/s (41.6MB/s), 39.7MiB/s-39.7MiB/s (41.6MB/s-41.6MB/s), io=278MiB (291MB), run=7001-7001msec
$ cat /tmp/fio-output_512_2_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751785,
"timestamp_ms" : 1666751785122,
"time" : "Wed Oct 26 04:36:25 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 922618368,
"io_kbytes" : 900994,
"bw_bytes" : 131783797,
"bw" : 128695,
"iops" : 257389.944294,
"runtime" : 7001,
"total_ios" : 1801987,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1583,
"max" : 29826,
"mean" : 2299.051626,
"stddev" : 743.697493,
"N" : 1801988
},
"clat_ns" : {
"min" : 230,
"max" : 138339,
"mean" : 5203.031010,
"stddev" : 1872.544552,
"N" : 1801987,
"percentile" : {
"1.000000" : 4128,
"5.000000" : 4192,
"10.000000" : 4192,
"20.000000" : 4256,
"30.000000" : 4320,
"40.000000" : 4448,
"50.000000" : 5408,
"60.000000" : 5472,
"70.000000" : 5600,
"80.000000" : 5664,
"90.000000" : 5856,
"95.000000" : 5984,
"99.000000" : 13376,
"99.500000" : 18048,
"99.900000" : 20864,
"99.950000" : 21120,
"99.990000" : 61184
}
},
"lat_ns" : {
"min" : 5740,
"max" : 140022,
"mean" : 7502.082299,
"stddev" : 1785.297905,
"N" : 1801987
},
"bw_min" : 128549,
"bw_max" : 129325,
"bw_agg" : 100.000000,
"bw_mean" : 128728.785714,
"bw_dev" : 240.665161,
"bw_samples" : 14,
"iops_min" : 257098,
"iops_max" : 258650,
"iops_mean" : 257457.714286,
"iops_stddev" : 481.220792,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 59.585714,
"sys_cpu" : 40.385714,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 1.640911,
"2" : 98.359089,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.011876,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 97.912249,
"20" : 1.800734,
"50" : 0.250335,
"100" : 0.017148,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=141271: Wed Oct 26 04:36:25 2022
read: IOPS=257k, BW=126MiB/s (132MB/s)(880MiB/7001msec)
slat (nsec): min=1583, max=29826, avg=2299.05, stdev=743.70
clat (nsec): min=230, max=138339, avg=5203.03, stdev=1872.54
lat (usec): min=5, max=140, avg= 7.50, stdev= 1.79
clat percentiles (nsec):
| 1.00th=[ 4128], 5.00th=[ 4192], 10.00th=[ 4192], 20.00th=[ 4256],
| 30.00th=[ 4320], 40.00th=[ 4448], 50.00th=[ 5408], 60.00th=[ 5472],
| 70.00th=[ 5600], 80.00th=[ 5664], 90.00th=[ 5856], 95.00th=[ 5984],
| 99.00th=[13376], 99.50th=[18048], 99.90th=[20864], 99.95th=[21120],
| 99.99th=[61184]
bw ( KiB/s): min=128549, max=129325, per=100.00%, avg=128728.79, stdev=240.67, samples=14
iops : min=257098, max=258650, avg=257457.71, stdev=481.22, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=97.91%, 20=1.80%, 50=0.25%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=59.59%, sys=40.39%, ctx=23, majf=0, minf=0
IO depths : 1=1.6%, 2=98.4%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1801987,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=880MiB (923MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751800,
"timestamp_ms" : 1666751800261,
"time" : "Wed Oct 26 04:36:40 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 892137472,
"io_kbytes" : 871228,
"bw_bytes" : 127430005,
"bw" : 124443,
"iops" : 248886.444794,
"runtime" : 7001,
"total_ios" : 1742454,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1593,
"max" : 19317,
"mean" : 2306.401335,
"stddev" : 734.829566,
"N" : 1742455
},
"clat_ns" : {
"min" : 591,
"max" : 121829,
"mean" : 5472.362839,
"stddev" : 1813.157116,
"N" : 1742454,
"percentile" : {
"1.000000" : 4448,
"5.000000" : 4512,
"10.000000" : 4512,
"20.000000" : 4576,
"30.000000" : 4576,
"40.000000" : 4640,
"50.000000" : 5728,
"60.000000" : 5792,
"70.000000" : 5856,
"80.000000" : 5920,
"90.000000" : 5984,
"95.000000" : 6176,
"99.000000" : 13248,
"99.500000" : 17536,
"99.900000" : 20352,
"99.950000" : 20864,
"99.990000" : 62208
}
},
"lat_ns" : {
"min" : 5901,
"max" : 123492,
"mean" : 7778.763796,
"stddev" : 1710.924115,
"N" : 1742454
},
"bw_min" : 124145,
"bw_max" : 124816,
"bw_agg" : 100.000000,
"bw_mean" : 124484.384615,
"bw_dev" : 207.810626,
"bw_samples" : 13,
"iops_min" : 248290,
"iops_max" : 249632,
"iops_mean" : 248968.692308,
"iops_stddev" : 415.787483,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.300000,
"sys_cpu" : 60.671429,
"ctx" : 18,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 1.483425,
"2" : 98.516575,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.013200,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.015098,
"20" : 1.779043,
"50" : 0.169072,
"100" : 0.019513,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141320: Wed Oct 26 04:36:40 2022
read: IOPS=249k, BW=122MiB/s (127MB/s)(851MiB/7001msec)
slat (nsec): min=1593, max=19317, avg=2306.40, stdev=734.83
clat (nsec): min=591, max=121829, avg=5472.36, stdev=1813.16
lat (usec): min=5, max=123, avg= 7.78, stdev= 1.71
clat percentiles (nsec):
| 1.00th=[ 4448], 5.00th=[ 4512], 10.00th=[ 4512], 20.00th=[ 4576],
| 30.00th=[ 4576], 40.00th=[ 4640], 50.00th=[ 5728], 60.00th=[ 5792],
| 70.00th=[ 5856], 80.00th=[ 5920], 90.00th=[ 5984], 95.00th=[ 6176],
| 99.00th=[13248], 99.50th=[17536], 99.90th=[20352], 99.95th=[20864],
| 99.99th=[62208]
bw ( KiB/s): min=124145, max=124816, per=100.00%, avg=124484.38, stdev=207.81, samples=13
iops : min=248290, max=249632, avg=248968.69, stdev=415.79, samples=13
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=98.02%, 20=1.78%, 50=0.17%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=39.30%, sys=60.67%, ctx=18, majf=0, minf=0
IO depths : 1=1.5%, 2=98.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1742454,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=122MiB/s (127MB/s), 122MiB/s-122MiB/s (127MB/s-127MB/s), io=851MiB (892MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751815,
"timestamp_ms" : 1666751815448,
"time" : "Wed Oct 26 04:36:55 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 888960000,
"io_kbytes" : 868125,
"bw_bytes" : 126976146,
"bw" : 124000,
"iops" : 248000.142837,
"runtime" : 7001,
"total_ios" : 1736249,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1613,
"max" : 33963,
"mean" : 2325.591072,
"stddev" : 759.135311,
"N" : 1736249
},
"clat_ns" : {
"min" : 621,
"max" : 121107,
"mean" : 5472.580111,
"stddev" : 1812.845915,
"N" : 1736250,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 4320,
"10.000000" : 4512,
"20.000000" : 4576,
"30.000000" : 4640,
"40.000000" : 4704,
"50.000000" : 5472,
"60.000000" : 5856,
"70.000000" : 5920,
"80.000000" : 5984,
"90.000000" : 6048,
"95.000000" : 6240,
"99.000000" : 13376,
"99.500000" : 17280,
"99.900000" : 20096,
"99.950000" : 20608,
"99.990000" : 60672
}
},
"lat_ns" : {
"min" : 5891,
"max" : 122811,
"mean" : 7798.170796,
"stddev" : 1715.249598,
"N" : 1736250
},
"bw_min" : 123725,
"bw_max" : 124153,
"bw_agg" : 100.000000,
"bw_mean" : 124029.571429,
"bw_dev" : 106.987213,
"bw_samples" : 14,
"iops_min" : 247450,
"iops_max" : 248306,
"iops_mean" : 248059.000000,
"iops_stddev" : 214.096671,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 35.585714,
"sys_cpu" : 64.385714,
"ctx" : 24,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 1.473003,
"2" : 98.526997,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.012268,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 97.911532,
"20" : 1.955998,
"50" : 0.096991,
"100" : 0.019410,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141369: Wed Oct 26 04:36:55 2022
read: IOPS=248k, BW=121MiB/s (127MB/s)(848MiB/7001msec)
slat (nsec): min=1613, max=33963, avg=2325.59, stdev=759.14
clat (nsec): min=621, max=121107, avg=5472.58, stdev=1812.85
lat (usec): min=5, max=122, avg= 7.80, stdev= 1.72
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 4320], 10.00th=[ 4512], 20.00th=[ 4576],
| 30.00th=[ 4640], 40.00th=[ 4704], 50.00th=[ 5472], 60.00th=[ 5856],
| 70.00th=[ 5920], 80.00th=[ 5984], 90.00th=[ 6048], 95.00th=[ 6240],
| 99.00th=[13376], 99.50th=[17280], 99.90th=[20096], 99.95th=[20608],
| 99.99th=[60672]
bw ( KiB/s): min=123725, max=124153, per=100.00%, avg=124029.57, stdev=106.99, samples=14
iops : min=247450, max=248306, avg=248059.00, stdev=214.10, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=97.91%, 20=1.96%, 50=0.10%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=35.59%, sys=64.39%, ctx=24, majf=0, minf=0
IO depths : 1=1.5%, 2=98.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1736249,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=848MiB (889MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751830,
"timestamp_ms" : 1666751830581,
"time" : "Wed Oct 26 04:37:10 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 886879232,
"io_kbytes" : 866093,
"bw_bytes" : 126678936,
"bw" : 123709,
"iops" : 247419.511498,
"runtime" : 7001,
"total_ios" : 1732184,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1623,
"max" : 26189,
"mean" : 2344.885200,
"stddev" : 746.431565,
"N" : 1732185
},
"clat_ns" : {
"min" : 621,
"max" : 175389,
"mean" : 5470.346042,
"stddev" : 1841.692980,
"N" : 1732184,
"percentile" : {
"1.000000" : 4256,
"5.000000" : 4320,
"10.000000" : 4512,
"20.000000" : 4576,
"30.000000" : 4640,
"40.000000" : 4704,
"50.000000" : 5536,
"60.000000" : 5856,
"70.000000" : 5920,
"80.000000" : 5984,
"90.000000" : 6048,
"95.000000" : 6240,
"99.000000" : 13120,
"99.500000" : 17024,
"99.900000" : 20096,
"99.950000" : 20352,
"99.990000" : 64768
}
},
"lat_ns" : {
"min" : 5921,
"max" : 177122,
"mean" : 7815.230866,
"stddev" : 1740.065131,
"N" : 1732184
},
"bw_min" : 123454,
"bw_max" : 124192,
"bw_agg" : 100.000000,
"bw_mean" : 123727.230769,
"bw_dev" : 248.715753,
"bw_samples" : 13,
"iops_min" : 246908,
"iops_max" : 248384,
"iops_mean" : 247454.461538,
"iops_stddev" : 497.431505,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 37.471429,
"sys_cpu" : 62.500000,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 1.468666,
"2" : 98.531334,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010045,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.048995,
"20" : 1.830464,
"50" : 0.086019,
"100" : 0.019109,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141419: Wed Oct 26 04:37:10 2022
read: IOPS=247k, BW=121MiB/s (127MB/s)(846MiB/7001msec)
slat (nsec): min=1623, max=26189, avg=2344.89, stdev=746.43
clat (nsec): min=621, max=175389, avg=5470.35, stdev=1841.69
lat (usec): min=5, max=177, avg= 7.82, stdev= 1.74
clat percentiles (nsec):
| 1.00th=[ 4256], 5.00th=[ 4320], 10.00th=[ 4512], 20.00th=[ 4576],
| 30.00th=[ 4640], 40.00th=[ 4704], 50.00th=[ 5536], 60.00th=[ 5856],
| 70.00th=[ 5920], 80.00th=[ 5984], 90.00th=[ 6048], 95.00th=[ 6240],
| 99.00th=[13120], 99.50th=[17024], 99.90th=[20096], 99.95th=[20352],
| 99.99th=[64768]
bw ( KiB/s): min=123454, max=124192, per=100.00%, avg=123727.23, stdev=248.72, samples=13
iops : min=246908, max=248384, avg=247454.46, stdev=497.43, samples=13
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=98.05%, 20=1.83%, 50=0.09%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=37.47%, sys=62.50%, ctx=22, majf=0, minf=0
IO depths : 1=1.5%, 2=98.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1732184,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=846MiB (887MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt
$ rm /tmp/fio-output_512_2_libaio-xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751844,
"timestamp_ms" : 1666751844361,
"time" : "Wed Oct 26 04:37:24 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 727684608,
"io_kbytes" : 710629,
"bw_bytes" : 103940095,
"bw" : 101503,
"iops" : 203007.713184,
"runtime" : 7001,
"total_ios" : 1421257,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1472,
"max" : 38092,
"mean" : 1673.767642,
"stddev" : 256.286057,
"N" : 1421258
},
"clat_ns" : {
"min" : 471,
"max" : 160200,
"mean" : 7924.788124,
"stddev" : 1919.052306,
"N" : 1421258,
"percentile" : {
"1.000000" : 4704,
"5.000000" : 7328,
"10.000000" : 7520,
"20.000000" : 7584,
"30.000000" : 7648,
"40.000000" : 7712,
"50.000000" : 7776,
"60.000000" : 7840,
"70.000000" : 7904,
"80.000000" : 7968,
"90.000000" : 8032,
"95.000000" : 8160,
"99.000000" : 15552,
"99.500000" : 19584,
"99.900000" : 25216,
"99.950000" : 25472,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5992,
"max" : 162054,
"mean" : 9598.555765,
"stddev" : 1912.374899,
"N" : 1421258
},
"bw_min" : 101229,
"bw_max" : 101973,
"bw_agg" : 100.000000,
"bw_mean" : 101532.571429,
"bw_dev" : 208.463136,
"bw_samples" : 14,
"iops_min" : 202458,
"iops_max" : 203946,
"iops_mean" : 203065.214286,
"iops_stddev" : 416.847542,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 22.600000,
"sys_cpu" : 57.742857,
"ctx" : 681640,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.858907,
"20" : 2.760233,
"50" : 0.346454,
"100" : 0.023078,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=141458: Wed Oct 26 04:37:24 2022
read: IOPS=203k, BW=99.1MiB/s (104MB/s)(694MiB/7001msec)
slat (nsec): min=1472, max=38092, avg=1673.77, stdev=256.29
clat (nsec): min=471, max=160200, avg=7924.79, stdev=1919.05
lat (usec): min=5, max=162, avg= 9.60, stdev= 1.91
clat percentiles (nsec):
| 1.00th=[ 4704], 5.00th=[ 7328], 10.00th=[ 7520], 20.00th=[ 7584],
| 30.00th=[ 7648], 40.00th=[ 7712], 50.00th=[ 7776], 60.00th=[ 7840],
| 70.00th=[ 7904], 80.00th=[ 7968], 90.00th=[ 8032], 95.00th=[ 8160],
| 99.00th=[15552], 99.50th=[19584], 99.90th=[25216], 99.95th=[25472],
| 99.99th=[68096]
bw ( KiB/s): min=101229, max=101973, per=100.00%, avg=101532.57, stdev=208.46, samples=14
iops : min=202458, max=203946, avg=203065.21, stdev=416.85, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 4=0.01%, 10=96.86%, 20=2.76%, 50=0.35%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=22.60%, sys=57.74%, ctx=681640, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1421257,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=99.1MiB/s (104MB/s), 99.1MiB/s-99.1MiB/s (104MB/s-104MB/s), io=694MiB (728MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751858,
"timestamp_ms" : 1666751858044,
"time" : "Wed Oct 26 04:37:38 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 724207104,
"io_kbytes" : 707233,
"bw_bytes" : 103443380,
"bw" : 101018,
"iops" : 202037.566062,
"runtime" : 7001,
"total_ios" : 1414465,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1472,
"max" : 29145,
"mean" : 1659.259898,
"stddev" : 223.201329,
"N" : 1414466
},
"clat_ns" : {
"min" : 451,
"max" : 122610,
"mean" : 7996.446592,
"stddev" : 1863.055498,
"N" : 1414466,
"percentile" : {
"1.000000" : 4896,
"5.000000" : 7520,
"10.000000" : 7584,
"20.000000" : 7584,
"30.000000" : 7648,
"40.000000" : 7712,
"50.000000" : 7840,
"60.000000" : 7840,
"70.000000" : 7904,
"80.000000" : 7968,
"90.000000" : 8032,
"95.000000" : 8256,
"99.000000" : 15680,
"99.500000" : 19584,
"99.900000" : 25472,
"99.950000" : 26240,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 5992,
"max" : 124454,
"mean" : 9655.706490,
"stddev" : 1864.008513,
"N" : 1414466
},
"bw_min" : 100860,
"bw_max" : 101305,
"bw_agg" : 100.000000,
"bw_mean" : 101042.214286,
"bw_dev" : 134.849305,
"bw_samples" : 14,
"iops_min" : 201720,
"iops_max" : 202610,
"iops_mean" : 202084.285714,
"iops_stddev" : 269.778094,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 22.514286,
"sys_cpu" : 57.471429,
"ctx" : 693903,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010888,
"10" : 96.823039,
"20" : 2.759206,
"50" : 0.380002,
"100" : 0.020432,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=141498: Wed Oct 26 04:37:38 2022
read: IOPS=202k, BW=98.7MiB/s (103MB/s)(691MiB/7001msec)
slat (nsec): min=1472, max=29145, avg=1659.26, stdev=223.20
clat (nsec): min=451, max=122610, avg=7996.45, stdev=1863.06
lat (usec): min=5, max=124, avg= 9.66, stdev= 1.86
clat percentiles (nsec):
| 1.00th=[ 4896], 5.00th=[ 7520], 10.00th=[ 7584], 20.00th=[ 7584],
| 30.00th=[ 7648], 40.00th=[ 7712], 50.00th=[ 7840], 60.00th=[ 7840],
| 70.00th=[ 7904], 80.00th=[ 7968], 90.00th=[ 8032], 95.00th=[ 8256],
| 99.00th=[15680], 99.50th=[19584], 99.90th=[25472], 99.95th=[26240],
| 99.99th=[67072]
bw ( KiB/s): min=100860, max=101305, per=100.00%, avg=101042.21, stdev=134.85, samples=14
iops : min=201720, max=202610, avg=202084.29, stdev=269.78, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 4=0.01%, 10=96.82%, 20=2.76%, 50=0.38%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=22.51%, sys=57.47%, ctx=693903, majf=0, minf=1
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1414465,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=98.7MiB/s (103MB/s), 98.7MiB/s-98.7MiB/s (103MB/s-103MB/s), io=691MiB (724MB), run=7001-7001msec
$ rm /tmp/fio-output_512_2_libaio-xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751871,
"timestamp_ms" : 1666751871710,
"time" : "Wed Oct 26 04:37:51 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "2",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 724407808,
"io_kbytes" : 707429,
"bw_bytes" : 103472047,
"bw" : 101046,
"iops" : 202093.700900,
"runtime" : 7001,
"total_ios" : 1414858,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1462,
"max" : 24455,
"mean" : 1634.391682,
"stddev" : 230.781785,
"N" : 1414858
},
"clat_ns" : {
"min" : 451,
"max" : 166613,
"mean" : 8011.332859,
"stddev" : 1891.589306,
"N" : 1414859,
"percentile" : {
"1.000000" : 4960,
"5.000000" : 7520,
"10.000000" : 7584,
"20.000000" : 7648,
"30.000000" : 7712,
"40.000000" : 7776,
"50.000000" : 7840,
"60.000000" : 7904,
"70.000000" : 7904,
"80.000000" : 7968,
"90.000000" : 8032,
"95.000000" : 8256,
"99.000000" : 15808,
"99.500000" : 19584,
"99.900000" : 25472,
"99.950000" : 25984,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 5981,
"max" : 168406,
"mean" : 9645.724596,
"stddev" : 1891.318592,
"N" : 1414859
},
"bw_min" : 100815,
"bw_max" : 101425,
"bw_agg" : 100.000000,
"bw_mean" : 101073.285714,
"bw_dev" : 170.786609,
"bw_samples" : 14,
"iops_min" : 201630,
"iops_max" : 202851,
"iops_mean" : 202146.642857,
"iops_stddev" : 341.731700,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.114286,
"sys_cpu" : 59.771429,
"ctx" : 694144,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 100.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 96.841308,
"20" : 2.744304,
"50" : 0.377635,
"100" : 0.023041,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 2,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=141538: Wed Oct 26 04:37:51 2022
read: IOPS=202k, BW=98.7MiB/s (103MB/s)(691MiB/7001msec)
slat (nsec): min=1462, max=24455, avg=1634.39, stdev=230.78
clat (nsec): min=451, max=166613, avg=8011.33, stdev=1891.59
lat (usec): min=5, max=168, avg= 9.65, stdev= 1.89
clat percentiles (nsec):
| 1.00th=[ 4960], 5.00th=[ 7520], 10.00th=[ 7584], 20.00th=[ 7648],
| 30.00th=[ 7712], 40.00th=[ 7776], 50.00th=[ 7840], 60.00th=[ 7904],
| 70.00th=[ 7904], 80.00th=[ 7968], 90.00th=[ 8032], 95.00th=[ 8256],
| 99.00th=[15808], 99.50th=[19584], 99.90th=[25472], 99.95th=[25984],
| 99.99th=[66048]
bw ( KiB/s): min=100815, max=101425, per=100.00%, avg=101073.29, stdev=170.79, samples=14
iops : min=201630, max=202851, avg=202146.64, stdev=341.73, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 4=0.01%, 10=96.84%, 20=2.74%, 50=0.38%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=20.11%, sys=59.77%, ctx=694144, majf=0, minf=0
IO depths : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1414858,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
Run status group 0 (all jobs):
READ: bw=98.7MiB/s (103MB/s), 98.7MiB/s-98.7MiB/s (103MB/s-103MB/s), io=691MiB (724MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751885,
"timestamp_ms" : 1666751885332,
"time" : "Wed Oct 26 04:38:05 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1191164928,
"io_kbytes" : 1163247,
"bw_bytes" : 170142112,
"bw" : 166154,
"iops" : 332308.241680,
"runtime" : 7001,
"total_ios" : 2326490,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1142,
"max" : 25858,
"mean" : 2197.265368,
"stddev" : 1099.926498,
"N" : 2326491
},
"clat_ns" : {
"min" : 80,
"max" : 165000,
"mean" : 9370.234935,
"stddev" : 2562.392231,
"N" : 2326493,
"percentile" : {
"1.000000" : 6048,
"5.000000" : 7648,
"10.000000" : 8640,
"20.000000" : 8768,
"30.000000" : 8768,
"40.000000" : 8768,
"50.000000" : 8896,
"60.000000" : 8896,
"70.000000" : 9024,
"80.000000" : 9152,
"90.000000" : 10304,
"95.000000" : 11584,
"99.000000" : 21888,
"99.500000" : 24192,
"99.900000" : 28800,
"99.950000" : 34048,
"99.990000" : 70144
}
},
"lat_ns" : {
"min" : 5481,
"max" : 166182,
"mean" : 11567.500667,
"stddev" : 2627.540346,
"N" : 2326493
},
"bw_min" : 165820,
"bw_max" : 166583,
"bw_agg" : 100.000000,
"bw_mean" : 166195.857143,
"bw_dev" : 234.095923,
"bw_samples" : 14,
"iops_min" : 331640,
"iops_max" : 333166,
"iops_mean" : 332391.785714,
"iops_stddev" : 468.114085,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.371429,
"sys_cpu" : 82.142857,
"ctx" : 13656,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.302774,
"10" : 82.745681,
"20" : 15.383217,
"50" : 1.505831,
"100" : 0.039759,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=141577: Wed Oct 26 04:38:05 2022
read: IOPS=332k, BW=162MiB/s (170MB/s)(1136MiB/7001msec)
slat (nsec): min=1142, max=25858, avg=2197.27, stdev=1099.93
clat (nsec): min=80, max=165000, avg=9370.23, stdev=2562.39
lat (usec): min=5, max=166, avg=11.57, stdev= 2.63
clat percentiles (nsec):
| 1.00th=[ 6048], 5.00th=[ 7648], 10.00th=[ 8640], 20.00th=[ 8768],
| 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8896], 60.00th=[ 8896],
| 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[10304], 95.00th=[11584],
| 99.00th=[21888], 99.50th=[24192], 99.90th=[28800], 99.95th=[34048],
| 99.99th=[70144]
bw ( KiB/s): min=165820, max=166583, per=100.00%, avg=166195.86, stdev=234.10, samples=14
iops : min=331640, max=333166, avg=332391.79, stdev=468.11, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.30%, 10=82.75%, 20=15.38%, 50=1.51%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=15.37%, sys=82.14%, ctx=13656, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2326490,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=1136MiB (1191MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751898,
"timestamp_ms" : 1666751898959,
"time" : "Wed Oct 26 04:38:18 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1195248640,
"io_kbytes" : 1167235,
"bw_bytes" : 170725416,
"bw" : 166724,
"iops" : 333447.507499,
"runtime" : 7001,
"total_ios" : 2334466,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1142,
"max" : 21070,
"mean" : 2160.286033,
"stddev" : 1053.646322,
"N" : 2334467
},
"clat_ns" : {
"min" : 70,
"max" : 172754,
"mean" : 9375.272461,
"stddev" : 2567.897705,
"N" : 2334469,
"percentile" : {
"1.000000" : 6048,
"5.000000" : 7584,
"10.000000" : 8640,
"20.000000" : 8768,
"30.000000" : 8768,
"40.000000" : 8768,
"50.000000" : 8896,
"60.000000" : 8896,
"70.000000" : 8896,
"80.000000" : 10048,
"90.000000" : 10304,
"95.000000" : 11584,
"99.000000" : 21888,
"99.500000" : 23936,
"99.900000" : 28288,
"99.950000" : 33536,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 5440,
"max" : 173937,
"mean" : 11535.557651,
"stddev" : 2590.223019,
"N" : 2334469
},
"bw_min" : 166368,
"bw_max" : 167296,
"bw_agg" : 100.000000,
"bw_mean" : 166781.142857,
"bw_dev" : 248.585493,
"bw_samples" : 14,
"iops_min" : 332737,
"iops_max" : 334592,
"iops_mean" : 333562.357143,
"iops_stddev" : 497.043197,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.000000,
"sys_cpu" : 82.514286,
"ctx" : 13822,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.012894,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.289659,
"10" : 79.779187,
"20" : 18.447945,
"50" : 1.422681,
"100" : 0.036282,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=141615: Wed Oct 26 04:38:18 2022
read: IOPS=333k, BW=163MiB/s (171MB/s)(1140MiB/7001msec)
slat (nsec): min=1142, max=21070, avg=2160.29, stdev=1053.65
clat (nsec): min=70, max=172754, avg=9375.27, stdev=2567.90
lat (usec): min=5, max=173, avg=11.54, stdev= 2.59
clat percentiles (nsec):
| 1.00th=[ 6048], 5.00th=[ 7584], 10.00th=[ 8640], 20.00th=[ 8768],
| 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8896], 60.00th=[ 8896],
| 70.00th=[ 8896], 80.00th=[10048], 90.00th=[10304], 95.00th=[11584],
| 99.00th=[21888], 99.50th=[23936], 99.90th=[28288], 99.95th=[33536],
| 99.99th=[72192]
bw ( KiB/s): min=166368, max=167296, per=100.00%, avg=166781.14, stdev=248.59, samples=14
iops : min=332737, max=334592, avg=333562.36, stdev=497.04, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.29%, 10=79.78%, 20=18.45%, 50=1.42%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=15.00%, sys=82.51%, ctx=13822, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2334466,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=1140MiB (1195MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751912,
"timestamp_ms" : 1666751912572,
"time" : "Wed Oct 26 04:38:32 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1193305600,
"io_kbytes" : 1165337,
"bw_bytes" : 170447878,
"bw" : 166453,
"iops" : 332905.442080,
"runtime" : 7001,
"total_ios" : 2330671,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1142,
"max" : 25989,
"mean" : 2188.620048,
"stddev" : 1059.038506,
"N" : 2330672
},
"clat_ns" : {
"min" : 70,
"max" : 194865,
"mean" : 9353.569892,
"stddev" : 2502.558563,
"N" : 2330674,
"percentile" : {
"1.000000" : 6048,
"5.000000" : 7584,
"10.000000" : 8640,
"20.000000" : 8768,
"30.000000" : 8768,
"40.000000" : 8768,
"50.000000" : 8896,
"60.000000" : 8896,
"70.000000" : 8896,
"80.000000" : 9280,
"90.000000" : 10304,
"95.000000" : 11584,
"99.000000" : 21888,
"99.500000" : 23680,
"99.900000" : 27264,
"99.950000" : 32640,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5451,
"max" : 196047,
"mean" : 11542.189115,
"stddev" : 2537.835066,
"N" : 2330674
},
"bw_min" : 166200,
"bw_max" : 167348,
"bw_agg" : 100.000000,
"bw_mean" : 166499.928571,
"bw_dev" : 288.296713,
"bw_samples" : 14,
"iops_min" : 332400,
"iops_max" : 334696,
"iops_mean" : 332999.857143,
"iops_stddev" : 576.593425,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.900000,
"sys_cpu" : 83.671429,
"ctx" : 13746,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.012014,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.293006,
"10" : 81.953008,
"20" : 16.269478,
"50" : 1.433021,
"100" : 0.029820,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=141653: Wed Oct 26 04:38:32 2022
read: IOPS=333k, BW=163MiB/s (170MB/s)(1138MiB/7001msec)
slat (nsec): min=1142, max=25989, avg=2188.62, stdev=1059.04
clat (nsec): min=70, max=194865, avg=9353.57, stdev=2502.56
lat (usec): min=5, max=196, avg=11.54, stdev= 2.54
clat percentiles (nsec):
| 1.00th=[ 6048], 5.00th=[ 7584], 10.00th=[ 8640], 20.00th=[ 8768],
| 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8896], 60.00th=[ 8896],
| 70.00th=[ 8896], 80.00th=[ 9280], 90.00th=[10304], 95.00th=[11584],
| 99.00th=[21888], 99.50th=[23680], 99.90th=[27264], 99.95th=[32640],
| 99.99th=[68096]
bw ( KiB/s): min=166200, max=167348, per=100.00%, avg=166499.93, stdev=288.30, samples=14
iops : min=332400, max=334696, avg=332999.86, stdev=576.59, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.29%, 10=81.95%, 20=16.27%, 50=1.43%
lat (usec) : 100=0.03%, 250=0.01%
cpu : usr=13.90%, sys=83.67%, ctx=13746, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2330671,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=163MiB/s (170MB/s), 163MiB/s-163MiB/s (170MB/s-170MB/s), io=1138MiB (1193MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751023,
"timestamp_ms" : 1666751023343,
"time" : "Wed Oct 26 04:23:43 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 477709312,
"io_kbytes" : 466513,
"bw_bytes" : 68234439,
"bw" : 66635,
"iops" : 133270.389944,
"runtime" : 7001,
"total_ios" : 933026,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 250,
"max" : 10320,
"mean" : 279.075861,
"stddev" : 23.844744,
"N" : 933026
},
"clat_ns" : {
"min" : 250,
"max" : 119153,
"mean" : 7007.570928,
"stddev" : 1331.921014,
"N" : 933026,
"percentile" : {
"1.000000" : 6624,
"5.000000" : 6688,
"10.000000" : 6752,
"20.000000" : 6752,
"30.000000" : 6752,
"40.000000" : 6816,
"50.000000" : 6816,
"60.000000" : 6880,
"70.000000" : 6880,
"80.000000" : 6944,
"90.000000" : 7008,
"95.000000" : 7136,
"99.000000" : 14400,
"99.500000" : 17024,
"99.900000" : 18048,
"99.950000" : 18560,
"99.990000" : 59136
}
},
"lat_ns" : {
"min" : 6392,
"max" : 119454,
"mean" : 7286.646789,
"stddev" : 1332.043757,
"N" : 933026
},
"bw_min" : 66542,
"bw_max" : 66739,
"bw_agg" : 100.000000,
"bw_mean" : 66649.214286,
"bw_dev" : 62.060986,
"bw_samples" : 14,
"iops_min" : 133084,
"iops_max" : 133478,
"iops_mean" : 133298.428571,
"iops_stddev" : 124.121971,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 27,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.287508,
"20" : 1.672622,
"50" : 0.025401,
"100" : 0.014148,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=138918: Wed Oct 26 04:23:43 2022
read: IOPS=133k, BW=65.1MiB/s (68.2MB/s)(456MiB/7001msec)
slat (nsec): min=250, max=10320, avg=279.08, stdev=23.84
clat (nsec): min=250, max=119153, avg=7007.57, stdev=1331.92
lat (usec): min=6, max=119, avg= 7.29, stdev= 1.33
clat percentiles (nsec):
| 1.00th=[ 6624], 5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6752],
| 30.00th=[ 6752], 40.00th=[ 6816], 50.00th=[ 6816], 60.00th=[ 6880],
| 70.00th=[ 6880], 80.00th=[ 6944], 90.00th=[ 7008], 95.00th=[ 7136],
| 99.00th=[14400], 99.50th=[17024], 99.90th=[18048], 99.95th=[18560],
| 99.99th=[59136]
bw ( KiB/s): min=66542, max=66739, per=100.00%, avg=66649.21, stdev=62.06, samples=14
iops : min=133084, max=133478, avg=133298.43, stdev=124.12, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=98.29%, 20=1.67%, 50=0.03%, 100=0.01%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=27, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=933026,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=65.1MiB/s (68.2MB/s), 65.1MiB/s-65.1MiB/s (68.2MB/s-68.2MB/s), io=456MiB (478MB), run=7001-7001msec
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751926,
"timestamp_ms" : 1666751926230,
"time" : "Wed Oct 26 04:38:46 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1147879936,
"io_kbytes" : 1120976,
"bw_bytes" : 163959425,
"bw" : 160116,
"iops" : 320232.681046,
"runtime" : 7001,
"total_ios" : 2241949,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1193,
"max" : 33182,
"mean" : 2542.839603,
"stddev" : 751.011231,
"N" : 2241950
},
"clat_ns" : {
"min" : 120,
"max" : 161323,
"mean" : 9665.963411,
"stddev" : 2557.083231,
"N" : 2241952,
"percentile" : {
"1.000000" : 6368,
"5.000000" : 8032,
"10.000000" : 9024,
"20.000000" : 9024,
"30.000000" : 9152,
"40.000000" : 9152,
"50.000000" : 9280,
"60.000000" : 9280,
"70.000000" : 9280,
"80.000000" : 9408,
"90.000000" : 10688,
"95.000000" : 10944,
"99.000000" : 22912,
"99.500000" : 24704,
"99.900000" : 28800,
"99.950000" : 33536,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 5630,
"max" : 162565,
"mean" : 12208.801880,
"stddev" : 2451.667220,
"N" : 2241952
},
"bw_min" : 159647,
"bw_max" : 160652,
"bw_agg" : 100.000000,
"bw_mean" : 160159.214286,
"bw_dev" : 275.571065,
"bw_samples" : 14,
"iops_min" : 319294,
"iops_max" : 321304,
"iops_mean" : 320318.500000,
"iops_stddev" : 551.411999,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.742857,
"sys_cpu" : 78.857143,
"ctx" : 12881,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.304556,
"10" : 85.096405,
"20" : 12.938698,
"50" : 1.610117,
"100" : 0.036888,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=141691: Wed Oct 26 04:38:46 2022
read: IOPS=320k, BW=156MiB/s (164MB/s)(1095MiB/7001msec)
slat (nsec): min=1193, max=33182, avg=2542.84, stdev=751.01
clat (nsec): min=120, max=161323, avg=9665.96, stdev=2557.08
lat (usec): min=5, max=162, avg=12.21, stdev= 2.45
clat percentiles (nsec):
| 1.00th=[ 6368], 5.00th=[ 8032], 10.00th=[ 9024], 20.00th=[ 9024],
| 30.00th=[ 9152], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9280],
| 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[10688], 95.00th=[10944],
| 99.00th=[22912], 99.50th=[24704], 99.90th=[28800], 99.95th=[33536],
| 99.99th=[71168]
bw ( KiB/s): min=159647, max=160652, per=100.00%, avg=160159.21, stdev=275.57, samples=14
iops : min=319294, max=321304, avg=320318.50, stdev=551.41, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.30%, 10=85.10%, 20=12.94%, 50=1.61%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=18.74%, sys=78.86%, ctx=12881, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2241949,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=1095MiB (1148MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751939,
"timestamp_ms" : 1666751939860,
"time" : "Wed Oct 26 04:38:59 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1146219008,
"io_kbytes" : 1119354,
"bw_bytes" : 163722183,
"bw" : 159884,
"iops" : 319769.318669,
"runtime" : 7001,
"total_ios" : 2238705,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1202,
"max" : 26811,
"mean" : 2540.398739,
"stddev" : 740.364198,
"N" : 2238706
},
"clat_ns" : {
"min" : 110,
"max" : 124784,
"mean" : 9685.766814,
"stddev" : 2569.003445,
"N" : 2238708,
"percentile" : {
"1.000000" : 6432,
"5.000000" : 8096,
"10.000000" : 9024,
"20.000000" : 9024,
"30.000000" : 9152,
"40.000000" : 9152,
"50.000000" : 9280,
"60.000000" : 9280,
"70.000000" : 9280,
"80.000000" : 9408,
"90.000000" : 10688,
"95.000000" : 10944,
"99.000000" : 23168,
"99.500000" : 24704,
"99.900000" : 29568,
"99.950000" : 34048,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 5580,
"max" : 126016,
"mean" : 12226.165602,
"stddev" : 2465.670806,
"N" : 2238708
},
"bw_min" : 159323,
"bw_max" : 160812,
"bw_agg" : 100.000000,
"bw_mean" : 159930.214286,
"bw_dev" : 504.450833,
"bw_samples" : 14,
"iops_min" : 318646,
"iops_max" : 321625,
"iops_mean" : 319860.500000,
"iops_stddev" : 1009.131595,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.500000,
"sys_cpu" : 82.100000,
"ctx" : 12883,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.293429,
"10" : 84.859327,
"20" : 13.183113,
"50" : 1.610619,
"100" : 0.040336,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=141729: Wed Oct 26 04:38:59 2022
read: IOPS=320k, BW=156MiB/s (164MB/s)(1093MiB/7001msec)
slat (nsec): min=1202, max=26811, avg=2540.40, stdev=740.36
clat (nsec): min=110, max=124784, avg=9685.77, stdev=2569.00
lat (usec): min=5, max=126, avg=12.23, stdev= 2.47
clat percentiles (nsec):
| 1.00th=[ 6432], 5.00th=[ 8096], 10.00th=[ 9024], 20.00th=[ 9024],
| 30.00th=[ 9152], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9280],
| 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[10688], 95.00th=[10944],
| 99.00th=[23168], 99.50th=[24704], 99.90th=[29568], 99.95th=[34048],
| 99.99th=[72192]
bw ( KiB/s): min=159323, max=160812, per=100.00%, avg=159930.21, stdev=504.45, samples=14
iops : min=318646, max=321625, avg=319860.50, stdev=1009.13, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.29%, 10=84.86%, 20=13.18%, 50=1.61%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=15.50%, sys=82.10%, ctx=12883, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2238705,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=1093MiB (1146MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751953,
"timestamp_ms" : 1666751953498,
"time" : "Wed Oct 26 04:39:13 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1122532864,
"io_kbytes" : 1096223,
"bw_bytes" : 160338932,
"bw" : 156580,
"iops" : 313161.405513,
"runtime" : 7001,
"total_ios" : 2192443,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1212,
"max" : 24466,
"mean" : 2454.161673,
"stddev" : 961.853907,
"N" : 2192444
},
"clat_ns" : {
"min" : 130,
"max" : 143108,
"mean" : 9983.846668,
"stddev" : 2529.342162,
"N" : 2192446,
"percentile" : {
"1.000000" : 7648,
"5.000000" : 8160,
"10.000000" : 9280,
"20.000000" : 9280,
"30.000000" : 9408,
"40.000000" : 9408,
"50.000000" : 9536,
"60.000000" : 9536,
"70.000000" : 9536,
"80.000000" : 10688,
"90.000000" : 10944,
"95.000000" : 11072,
"99.000000" : 22912,
"99.500000" : 24704,
"99.900000" : 28544,
"99.950000" : 33536,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 5500,
"max" : 144361,
"mean" : 12438.008588,
"stddev" : 2576.508476,
"N" : 2192446
},
"bw_min" : 156329,
"bw_max" : 156855,
"bw_agg" : 100.000000,
"bw_mean" : 156627.857143,
"bw_dev" : 180.425078,
"bw_samples" : 14,
"iops_min" : 312658,
"iops_max" : 313710,
"iops_mean" : 313255.714286,
"iops_stddev" : 361.150177,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 20.500000,
"sys_cpu" : 77.171429,
"ctx" : 12638,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.095647,
"10" : 78.931676,
"20" : 19.299658,
"50" : 1.626907,
"100" : 0.036307,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=141767: Wed Oct 26 04:39:13 2022
read: IOPS=313k, BW=153MiB/s (160MB/s)(1071MiB/7001msec)
slat (nsec): min=1212, max=24466, avg=2454.16, stdev=961.85
clat (nsec): min=130, max=143108, avg=9983.85, stdev=2529.34
lat (usec): min=5, max=144, avg=12.44, stdev= 2.58
clat percentiles (nsec):
| 1.00th=[ 7648], 5.00th=[ 8160], 10.00th=[ 9280], 20.00th=[ 9280],
| 30.00th=[ 9408], 40.00th=[ 9408], 50.00th=[ 9536], 60.00th=[ 9536],
| 70.00th=[ 9536], 80.00th=[10688], 90.00th=[10944], 95.00th=[11072],
| 99.00th=[22912], 99.50th=[24704], 99.90th=[28544], 99.95th=[33536],
| 99.99th=[71168]
bw ( KiB/s): min=156329, max=156855, per=100.00%, avg=156627.86, stdev=180.43, samples=14
iops : min=312658, max=313710, avg=313255.71, stdev=361.15, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.10%, 10=78.93%, 20=19.30%, 50=1.63%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=20.50%, sys=77.17%, ctx=12638, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2192443,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=1071MiB (1123MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751968,
"timestamp_ms" : 1666751968527,
"time" : "Wed Oct 26 04:39:28 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1505933312,
"io_kbytes" : 1470638,
"bw_bytes" : 215102601,
"bw" : 210061,
"iops" : 420121.839737,
"runtime" : 7001,
"total_ios" : 2941273,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 210,
"max" : 28404,
"mean" : 272.227385,
"stddev" : 44.424831,
"N" : 2941273
},
"clat_ns" : {
"min" : 360,
"max" : 125015,
"mean" : 9017.813720,
"stddev" : 2595.881437,
"N" : 2941276,
"percentile" : {
"1.000000" : 6752,
"5.000000" : 7456,
"10.000000" : 7840,
"20.000000" : 7968,
"30.000000" : 8032,
"40.000000" : 8160,
"50.000000" : 8256,
"60.000000" : 9152,
"70.000000" : 9280,
"80.000000" : 9408,
"90.000000" : 9664,
"95.000000" : 10560,
"99.000000" : 20352,
"99.500000" : 22144,
"99.900000" : 29568,
"99.950000" : 36096,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 5751,
"max" : 125285,
"mean" : 9290.041100,
"stddev" : 2596.346238,
"N" : 2941276
},
"bw_min" : 209329,
"bw_max" : 211028,
"bw_agg" : 100.000000,
"bw_mean" : 210143.428571,
"bw_dev" : 467.370338,
"bw_samples" : 14,
"iops_min" : 418658,
"iops_max" : 422056,
"iops_mean" : 420286.428571,
"iops_stddev" : 934.806662,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 21,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 1.919441,
"4" : 98.072977,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 93.588456,
"20" : 5.184796,
"50" : 1.186323,
"100" : 0.035801,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141818: Wed Oct 26 04:39:28 2022
read: IOPS=420k, BW=205MiB/s (215MB/s)(1436MiB/7001msec)
slat (nsec): min=210, max=28404, avg=272.23, stdev=44.42
clat (nsec): min=360, max=125015, avg=9017.81, stdev=2595.88
lat (usec): min=5, max=125, avg= 9.29, stdev= 2.60
clat percentiles (nsec):
| 1.00th=[ 6752], 5.00th=[ 7456], 10.00th=[ 7840], 20.00th=[ 7968],
| 30.00th=[ 8032], 40.00th=[ 8160], 50.00th=[ 8256], 60.00th=[ 9152],
| 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9664], 95.00th=[10560],
| 99.00th=[20352], 99.50th=[22144], 99.90th=[29568], 99.95th=[36096],
| 99.99th=[69120]
bw ( KiB/s): min=209329, max=211028, per=100.00%, avg=210143.43, stdev=467.37, samples=14
iops : min=418658, max=422056, avg=420286.43, stdev=934.81, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=93.59%, 20=5.18%, 50=1.19%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=21, majf=0, minf=0
IO depths : 1=0.1%, 2=1.9%, 4=98.1%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2941273,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=1436MiB (1506MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory
$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751983,
"timestamp_ms" : 1666751983603,
"time" : "Wed Oct 26 04:39:43 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1417665024,
"io_kbytes" : 1384438,
"bw_bytes" : 202494647,
"bw" : 197748,
"iops" : 395496.929010,
"runtime" : 7001,
"total_ios" : 2768874,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 170,
"max" : 8436,
"mean" : 264.949002,
"stddev" : 22.607211,
"N" : 2768874
},
"clat_ns" : {
"min" : 2355,
"max" : 122710,
"mean" : 9554.498064,
"stddev" : 2412.332312,
"N" : 2768877,
"percentile" : {
"1.000000" : 6432,
"5.000000" : 7392,
"10.000000" : 8032,
"20.000000" : 8768,
"30.000000" : 9152,
"40.000000" : 9152,
"50.000000" : 9280,
"60.000000" : 9408,
"70.000000" : 9536,
"80.000000" : 9792,
"90.000000" : 10432,
"95.000000" : 11840,
"99.000000" : 20096,
"99.500000" : 22144,
"99.900000" : 28032,
"99.950000" : 35072,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 5390,
"max" : 122971,
"mean" : 9819.447028,
"stddev" : 2413.593380,
"N" : 2768877
},
"bw_min" : 197589,
"bw_max" : 198348,
"bw_agg" : 100.000000,
"bw_mean" : 197841.000000,
"bw_dev" : 217.461933,
"bw_samples" : 14,
"iops_min" : 395178,
"iops_max" : 396696,
"iops_mean" : 395682.071429,
"iops_stddev" : 435.142499,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 24,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 11.586082,
"4" : 88.411318,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 87.145894,
"20" : 11.771283,
"50" : 1.041795,
"100" : 0.036007,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141869: Wed Oct 26 04:39:43 2022
read: IOPS=395k, BW=193MiB/s (202MB/s)(1352MiB/7001msec)
slat (nsec): min=170, max=8436, avg=264.95, stdev=22.61
clat (usec): min=2, max=122, avg= 9.55, stdev= 2.41
lat (usec): min=5, max=122, avg= 9.82, stdev= 2.41
clat percentiles (nsec):
| 1.00th=[ 6432], 5.00th=[ 7392], 10.00th=[ 8032], 20.00th=[ 8768],
| 30.00th=[ 9152], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9408],
| 70.00th=[ 9536], 80.00th=[ 9792], 90.00th=[10432], 95.00th=[11840],
| 99.00th=[20096], 99.50th=[22144], 99.90th=[28032], 99.95th=[35072],
| 99.99th=[69120]
bw ( KiB/s): min=197589, max=198348, per=100.00%, avg=197841.00, stdev=217.46, samples=14
iops : min=395178, max=396696, avg=395682.07, stdev=435.14, samples=14
lat (usec) : 4=0.01%, 10=87.15%, 20=11.77%, 50=1.04%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=24, majf=0, minf=0
IO depths : 1=0.1%, 2=11.6%, 4=88.4%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2768874,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=193MiB/s (202MB/s), 193MiB/s-193MiB/s (202MB/s-202MB/s), io=1352MiB (1418MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751998,
"timestamp_ms" : 1666751998687,
"time" : "Wed Oct 26 04:39:58 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1425394688,
"io_kbytes" : 1391987,
"bw_bytes" : 203598727,
"bw" : 198826,
"iops" : 397653.335238,
"runtime" : 7001,
"total_ios" : 2783971,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 180,
"max" : 14417,
"mean" : 263.192554,
"stddev" : 26.016764,
"N" : 2783971
},
"clat_ns" : {
"min" : 391,
"max" : 156223,
"mean" : 9514.911947,
"stddev" : 2418.859053,
"N" : 2783974,
"percentile" : {
"1.000000" : 6432,
"5.000000" : 7456,
"10.000000" : 8160,
"20.000000" : 8768,
"30.000000" : 9024,
"40.000000" : 9152,
"50.000000" : 9280,
"60.000000" : 9280,
"70.000000" : 9408,
"80.000000" : 9536,
"90.000000" : 10176,
"95.000000" : 11584,
"99.000000" : 20352,
"99.500000" : 22144,
"99.900000" : 27776,
"99.950000" : 34560,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 5100,
"max" : 156484,
"mean" : 9778.104498,
"stddev" : 2419.649924,
"N" : 2783974
},
"bw_min" : 198533,
"bw_max" : 199591,
"bw_agg" : 100.000000,
"bw_mean" : 198952.285714,
"bw_dev" : 317.459247,
"bw_samples" : 14,
"iops_min" : 397066,
"iops_max" : 399182,
"iops_mean" : 397904.714286,
"iops_stddev" : 634.979519,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 16,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 7.956728,
"4" : 92.041656,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 88.694997,
"20" : 10.182757,
"50" : 1.083416,
"100" : 0.033872,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141919: Wed Oct 26 04:39:58 2022
read: IOPS=398k, BW=194MiB/s (204MB/s)(1359MiB/7001msec)
slat (nsec): min=180, max=14417, avg=263.19, stdev=26.02
clat (nsec): min=391, max=156223, avg=9514.91, stdev=2418.86
lat (usec): min=5, max=156, avg= 9.78, stdev= 2.42
clat percentiles (nsec):
| 1.00th=[ 6432], 5.00th=[ 7456], 10.00th=[ 8160], 20.00th=[ 8768],
| 30.00th=[ 9024], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9280],
| 70.00th=[ 9408], 80.00th=[ 9536], 90.00th=[10176], 95.00th=[11584],
| 99.00th=[20352], 99.50th=[22144], 99.90th=[27776], 99.95th=[34560],
| 99.99th=[69120]
bw ( KiB/s): min=198533, max=199591, per=100.00%, avg=198952.29, stdev=317.46, samples=14
iops : min=397066, max=399182, avg=397904.71, stdev=634.98, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=88.69%, 20=10.18%, 50=1.08%, 100=0.03%
lat (usec) : 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=16, majf=0, minf=0
IO depths : 1=0.1%, 2=8.0%, 4=92.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2783971,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=194MiB/s (204MB/s), 194MiB/s-194MiB/s (204MB/s-204MB/s), io=1359MiB (1425MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752012,
"timestamp_ms" : 1666752012376,
"time" : "Wed Oct 26 04:40:12 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1110281216,
"io_kbytes" : 1084259,
"bw_bytes" : 158588946,
"bw" : 154872,
"iops" : 309743.465219,
"runtime" : 7001,
"total_ios" : 2168514,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1292,
"max" : 24085,
"mean" : 2504.110085,
"stddev" : 952.888239,
"N" : 2168515
},
"clat_ns" : {
"min" : 80,
"max" : 198122,
"mean" : 10032.524192,
"stddev" : 2637.425346,
"N" : 2168517,
"percentile" : {
"1.000000" : 7520,
"5.000000" : 8032,
"10.000000" : 9280,
"20.000000" : 9408,
"30.000000" : 9408,
"40.000000" : 9408,
"50.000000" : 9536,
"60.000000" : 9536,
"70.000000" : 9664,
"80.000000" : 10688,
"90.000000" : 11072,
"95.000000" : 11328,
"99.000000" : 23168,
"99.500000" : 24960,
"99.900000" : 29312,
"99.950000" : 34560,
"99.990000" : 73216
}
},
"lat_ns" : {
"min" : 5731,
"max" : 199454,
"mean" : 12536.634481,
"stddev" : 2666.909686,
"N" : 2168517
},
"bw_min" : 154763,
"bw_max" : 155474,
"bw_agg" : 100.000000,
"bw_mean" : 154943.461538,
"bw_dev" : 227.947953,
"bw_samples" : 13,
"iops_min" : 309526,
"iops_max" : 310949,
"iops_mean" : 309887.000000,
"iops_stddev" : 456.089903,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.742857,
"sys_cpu" : 82.900000,
"ctx" : 12590,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.086188,
"10" : 78.649711,
"20" : 19.549194,
"50" : 1.657863,
"100" : 0.040166,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 3043416,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 15511,
"write_ticks" : 0,
"in_queue" : 15511,
"util" : 99.035239
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=141958: Wed Oct 26 04:40:12 2022
read: IOPS=310k, BW=151MiB/s (159MB/s)(1059MiB/7001msec)
slat (nsec): min=1292, max=24085, avg=2504.11, stdev=952.89
clat (nsec): min=80, max=198122, avg=10032.52, stdev=2637.43
lat (usec): min=5, max=199, avg=12.54, stdev= 2.67
clat percentiles (nsec):
| 1.00th=[ 7520], 5.00th=[ 8032], 10.00th=[ 9280], 20.00th=[ 9408],
| 30.00th=[ 9408], 40.00th=[ 9408], 50.00th=[ 9536], 60.00th=[ 9536],
| 70.00th=[ 9664], 80.00th=[10688], 90.00th=[11072], 95.00th=[11328],
| 99.00th=[23168], 99.50th=[24960], 99.90th=[29312], 99.95th=[34560],
| 99.99th=[73216]
bw ( KiB/s): min=154763, max=155474, per=100.00%, avg=154943.46, stdev=227.95, samples=13
iops : min=309526, max=310949, avg=309887.00, stdev=456.09, samples=13
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.09%, 10=78.65%, 20=19.55%, 50=1.66%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=14.74%, sys=82.90%, ctx=12590, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2168514,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=151MiB/s (159MB/s), 151MiB/s-151MiB/s (159MB/s-159MB/s), io=1059MiB (1110MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=3043416/0, merge=0/0, ticks=15511/0, in_queue=15511, util=99.04%
$ rm /tmp/fio-output_512_4_io_uring-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752025,
"timestamp_ms" : 1666752025990,
"time" : "Wed Oct 26 04:40:25 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1115557888,
"io_kbytes" : 1089412,
"bw_bytes" : 159342649,
"bw" : 155608,
"iops" : 311215.540637,
"runtime" : 7001,
"total_ios" : 2178820,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1292,
"max" : 28123,
"mean" : 2499.686404,
"stddev" : 889.896990,
"N" : 2178821
},
"clat_ns" : {
"min" : 70,
"max" : 142838,
"mean" : 9952.938270,
"stddev" : 2613.804392,
"N" : 2178823,
"percentile" : {
"1.000000" : 7392,
"5.000000" : 8032,
"10.000000" : 9152,
"20.000000" : 9280,
"30.000000" : 9408,
"40.000000" : 9408,
"50.000000" : 9536,
"60.000000" : 9536,
"70.000000" : 9536,
"80.000000" : 10432,
"90.000000" : 11072,
"95.000000" : 11328,
"99.000000" : 23168,
"99.500000" : 24960,
"99.900000" : 29824,
"99.950000" : 34560,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 5541,
"max" : 144180,
"mean" : 12452.624922,
"stddev" : 2606.970828,
"N" : 2178823
},
"bw_min" : 155411,
"bw_max" : 156036,
"bw_agg" : 100.000000,
"bw_mean" : 155680.285714,
"bw_dev" : 179.364017,
"bw_samples" : 14,
"iops_min" : 310822,
"iops_max" : 312072,
"iops_mean" : 311360.571429,
"iops_stddev" : 358.728034,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.085714,
"sys_cpu" : 82.542857,
"ctx" : 12650,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010281,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.110702,
"10" : 79.515885,
"20" : 18.645368,
"50" : 1.666728,
"100" : 0.042362,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 3132143,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 15922,
"write_ticks" : 0,
"in_queue" : 15922,
"util" : 99.098564
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=141996: Wed Oct 26 04:40:25 2022
read: IOPS=311k, BW=152MiB/s (159MB/s)(1064MiB/7001msec)
slat (nsec): min=1292, max=28123, avg=2499.69, stdev=889.90
clat (nsec): min=70, max=142838, avg=9952.94, stdev=2613.80
lat (usec): min=5, max=144, avg=12.45, stdev= 2.61
clat percentiles (nsec):
| 1.00th=[ 7392], 5.00th=[ 8032], 10.00th=[ 9152], 20.00th=[ 9280],
| 30.00th=[ 9408], 40.00th=[ 9408], 50.00th=[ 9536], 60.00th=[ 9536],
| 70.00th=[ 9536], 80.00th=[10432], 90.00th=[11072], 95.00th=[11328],
| 99.00th=[23168], 99.50th=[24960], 99.90th=[29824], 99.95th=[34560],
| 99.99th=[72192]
bw ( KiB/s): min=155411, max=156036, per=100.00%, avg=155680.29, stdev=179.36, samples=14
iops : min=310822, max=312072, avg=311360.57, stdev=358.73, samples=14
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.11%, 10=79.52%, 20=18.65%, 50=1.67%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=15.09%, sys=82.54%, ctx=12650, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2178820,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=1064MiB (1116MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=3132143/0, merge=0/0, ticks=15922/0, in_queue=15922, util=99.10%
$ rm /tmp/fio-output_512_4_io_uring-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752039,
"timestamp_ms" : 1666752039618,
"time" : "Wed Oct 26 04:40:39 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1114947584,
"io_kbytes" : 1088816,
"bw_bytes" : 159255475,
"bw" : 155522,
"iops" : 311045.279246,
"runtime" : 7001,
"total_ios" : 2177628,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1282,
"max" : 115326,
"mean" : 2502.982368,
"stddev" : 919.655007,
"N" : 2177629
},
"clat_ns" : {
"min" : 70,
"max" : 153448,
"mean" : 9965.891609,
"stddev" : 2636.066476,
"N" : 2177631,
"percentile" : {
"1.000000" : 7456,
"5.000000" : 8032,
"10.000000" : 9152,
"20.000000" : 9280,
"30.000000" : 9408,
"40.000000" : 9408,
"50.000000" : 9536,
"60.000000" : 9536,
"70.000000" : 9536,
"80.000000" : 9920,
"90.000000" : 11072,
"95.000000" : 11200,
"99.000000" : 23168,
"99.500000" : 24960,
"99.900000" : 29312,
"99.950000" : 34048,
"99.990000" : 73216
}
},
"lat_ns" : {
"min" : 5721,
"max" : 154770,
"mean" : 12468.873560,
"stddev" : 2639.118189,
"N" : 2177631
},
"bw_min" : 155227,
"bw_max" : 156354,
"bw_agg" : 100.000000,
"bw_mean" : 155585.285714,
"bw_dev" : 335.643453,
"bw_samples" : 14,
"iops_min" : 310454,
"iops_max" : 312710,
"iops_mean" : 311170.571429,
"iops_stddev" : 671.748776,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.314286,
"sys_cpu" : 82.328571,
"ctx" : 12629,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.015154,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.107319,
"10" : 79.928757,
"20" : 18.224600,
"50" : 1.676457,
"100" : 0.037380,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 3131615,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 15931,
"write_ticks" : 0,
"in_queue" : 15931,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=142034: Wed Oct 26 04:40:39 2022
read: IOPS=311k, BW=152MiB/s (159MB/s)(1063MiB/7001msec)
slat (nsec): min=1282, max=115326, avg=2502.98, stdev=919.66
clat (nsec): min=70, max=153448, avg=9965.89, stdev=2636.07
lat (usec): min=5, max=154, avg=12.47, stdev= 2.64
clat percentiles (nsec):
| 1.00th=[ 7456], 5.00th=[ 8032], 10.00th=[ 9152], 20.00th=[ 9280],
| 30.00th=[ 9408], 40.00th=[ 9408], 50.00th=[ 9536], 60.00th=[ 9536],
| 70.00th=[ 9536], 80.00th=[ 9920], 90.00th=[11072], 95.00th=[11200],
| 99.00th=[23168], 99.50th=[24960], 99.90th=[29312], 99.95th=[34048],
| 99.99th=[73216]
bw ( KiB/s): min=155227, max=156354, per=100.00%, avg=155585.29, stdev=335.64, samples=14
iops : min=310454, max=312710, avg=311170.57, stdev=671.75, samples=14
lat (nsec) : 100=0.02%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.11%, 10=79.93%, 20=18.22%, 50=1.68%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=15.31%, sys=82.33%, ctx=12629, majf=0, minf=1
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2177628,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=1063MiB (1115MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=3131615/0, merge=0/0, ticks=15931/0, in_queue=15931, util=99.09%
$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752054,
"timestamp_ms" : 1666752054612,
"time" : "Wed Oct 26 04:40:54 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1012919808,
"io_kbytes" : 989179,
"bw_bytes" : 144682160,
"bw" : 141291,
"iops" : 282581.916869,
"runtime" : 7001,
"total_ios" : 1978356,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 26279,
"mean" : 141.316532,
"stddev" : 49.171458,
"N" : 1978356
},
"clat_ns" : {
"min" : 11882,
"max" : 177744,
"mean" : 13172.391741,
"stddev" : 2083.866987,
"N" : 1978356,
"percentile" : {
"1.000000" : 12224,
"5.000000" : 12352,
"10.000000" : 12352,
"20.000000" : 12608,
"30.000000" : 12608,
"40.000000" : 12736,
"50.000000" : 12864,
"60.000000" : 12992,
"70.000000" : 13120,
"80.000000" : 13248,
"90.000000" : 13376,
"95.000000" : 14016,
"99.000000" : 22912,
"99.500000" : 23936,
"99.900000" : 30592,
"99.950000" : 46336,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 12003,
"max" : 177944,
"mean" : 13313.708273,
"stddev" : 2088.192140,
"N" : 1978356
},
"bw_min" : 141168,
"bw_max" : 141492,
"bw_agg" : 100.000000,
"bw_mean" : 141330.928571,
"bw_dev" : 101.098934,
"bw_samples" : 14,
"iops_min" : 282336,
"iops_max" : 282984,
"iops_mean" : 282661.928571,
"iops_stddev" : 201.869137,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 38.900000,
"sys_cpu" : 61.071429,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 25.000000,
"2" : 50.000000,
"4" : 25.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 98.583622,
"50" : 1.369925,
"100" : 0.039477,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=142084: Wed Oct 26 04:40:54 2022
read: IOPS=283k, BW=138MiB/s (145MB/s)(966MiB/7001msec)
slat (nsec): min=110, max=26279, avg=141.32, stdev=49.17
clat (usec): min=11, max=177, avg=13.17, stdev= 2.08
lat (usec): min=12, max=177, avg=13.31, stdev= 2.09
clat percentiles (nsec):
| 1.00th=[12224], 5.00th=[12352], 10.00th=[12352], 20.00th=[12608],
| 30.00th=[12608], 40.00th=[12736], 50.00th=[12864], 60.00th=[12992],
| 70.00th=[13120], 80.00th=[13248], 90.00th=[13376], 95.00th=[14016],
| 99.00th=[22912], 99.50th=[23936], 99.90th=[30592], 99.95th=[46336],
| 99.99th=[71168]
bw ( KiB/s): min=141168, max=141492, per=100.00%, avg=141330.93, stdev=101.10, samples=14
iops : min=282336, max=282984, avg=282661.93, stdev=201.87, samples=14
lat (usec) : 20=98.58%, 50=1.37%, 100=0.04%, 250=0.01%
cpu : usr=38.90%, sys=61.07%, ctx=22, majf=0, minf=0
IO depths : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1978356,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=966MiB (1013MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751038,
"timestamp_ms" : 1666751038438,
"time" : "Wed Oct 26 04:23:58 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 438721024,
"io_kbytes" : 428438,
"bw_bytes" : 62665479,
"bw" : 61196,
"iops" : 122393.372375,
"runtime" : 7001,
"total_ios" : 856876,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 711,
"max" : 19336,
"mean" : 922.608556,
"stddev" : 65.998960,
"N" : 856877
},
"clat_ns" : {
"min" : 731,
"max" : 1292413,
"mean" : 6966.034914,
"stddev" : 1782.190180,
"N" : 856876,
"percentile" : {
"1.000000" : 6368,
"5.000000" : 6496,
"10.000000" : 6560,
"20.000000" : 6624,
"30.000000" : 6752,
"40.000000" : 6752,
"50.000000" : 6816,
"60.000000" : 6944,
"70.000000" : 6944,
"80.000000" : 7008,
"90.000000" : 7136,
"95.000000" : 7264,
"99.000000" : 12864,
"99.500000" : 13888,
"99.900000" : 16512,
"99.950000" : 16768,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 6993,
"max" : 1293966,
"mean" : 7888.643483,
"stddev" : 1784.824087,
"N" : 856876
},
"bw_min" : 61040,
"bw_max" : 61478,
"bw_agg" : 100.000000,
"bw_mean" : 61222.769231,
"bw_dev" : 152.265751,
"bw_samples" : 13,
"iops_min" : 122080,
"iops_max" : 122956,
"iops_mean" : 122445.769231,
"iops_stddev" : 304.518241,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 21,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.313175,
"20" : 1.668036,
"50" : 0.010000,
"100" : 0.013771,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=138968: Wed Oct 26 04:23:58 2022
read: IOPS=122k, BW=59.8MiB/s (62.7MB/s)(418MiB/7001msec)
slat (nsec): min=711, max=19336, avg=922.61, stdev=66.00
clat (nsec): min=731, max=1292.4k, avg=6966.03, stdev=1782.19
lat (usec): min=6, max=1293, avg= 7.89, stdev= 1.78
clat percentiles (nsec):
| 1.00th=[ 6368], 5.00th=[ 6496], 10.00th=[ 6560], 20.00th=[ 6624],
| 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6816], 60.00th=[ 6944],
| 70.00th=[ 6944], 80.00th=[ 7008], 90.00th=[ 7136], 95.00th=[ 7264],
| 99.00th=[12864], 99.50th=[13888], 99.90th=[16512], 99.95th=[16768],
| 99.99th=[58112]
bw ( KiB/s): min=61040, max=61478, per=100.00%, avg=61222.77, stdev=152.27, samples=13
iops : min=122080, max=122956, avg=122445.77, stdev=304.52, samples=13
lat (nsec) : 750=0.01%
lat (usec) : 10=98.31%, 20=1.67%, 50=0.01%, 100=0.01%
lat (msec) : 2=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=21, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=856876,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=59.8MiB/s (62.7MB/s), 59.8MiB/s-59.8MiB/s (62.7MB/s-62.7MB/s), io=418MiB (439MB), run=7001-7001msec
$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752069,
"timestamp_ms" : 1666752069718,
"time" : "Wed Oct 26 04:41:09 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1013155328,
"io_kbytes" : 989409,
"bw_bytes" : 144715801,
"bw" : 141324,
"iops" : 282647.621768,
"runtime" : 7001,
"total_ios" : 1978816,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 12503,
"mean" : 142.496506,
"stddev" : 47.180392,
"N" : 1978816
},
"clat_ns" : {
"min" : 11873,
"max" : 117370,
"mean" : 13154.827341,
"stddev" : 1841.054914,
"N" : 1978816,
"percentile" : {
"1.000000" : 12224,
"5.000000" : 12352,
"10.000000" : 12352,
"20.000000" : 12608,
"30.000000" : 12608,
"40.000000" : 12736,
"50.000000" : 12864,
"60.000000" : 12992,
"70.000000" : 13120,
"80.000000" : 13248,
"90.000000" : 13376,
"95.000000" : 13888,
"99.000000" : 22912,
"99.500000" : 23936,
"99.900000" : 29568,
"99.950000" : 35072,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 11993,
"max" : 117570,
"mean" : 13297.323847,
"stddev" : 1846.116285,
"N" : 1978816
},
"bw_min" : 141129,
"bw_max" : 141560,
"bw_agg" : 100.000000,
"bw_mean" : 141360.928571,
"bw_dev" : 153.096483,
"bw_samples" : 14,
"iops_min" : 282256,
"iops_max" : 283120,
"iops_mean" : 282722.000000,
"iops_stddev" : 306.117122,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 41.242857,
"sys_cpu" : 58.728571,
"ctx" : 20,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 25.000000,
"2" : 50.000000,
"4" : 25.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 98.730200,
"50" : 1.228361,
"100" : 0.041237,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=142133: Wed Oct 26 04:41:09 2022
read: IOPS=283k, BW=138MiB/s (145MB/s)(966MiB/7001msec)
slat (nsec): min=110, max=12503, avg=142.50, stdev=47.18
clat (usec): min=11, max=117, avg=13.15, stdev= 1.84
lat (usec): min=11, max=117, avg=13.30, stdev= 1.85
clat percentiles (nsec):
| 1.00th=[12224], 5.00th=[12352], 10.00th=[12352], 20.00th=[12608],
| 30.00th=[12608], 40.00th=[12736], 50.00th=[12864], 60.00th=[12992],
| 70.00th=[13120], 80.00th=[13248], 90.00th=[13376], 95.00th=[13888],
| 99.00th=[22912], 99.50th=[23936], 99.90th=[29568], 99.95th=[35072],
| 99.99th=[66048]
bw ( KiB/s): min=141129, max=141560, per=100.00%, avg=141360.93, stdev=153.10, samples=14
iops : min=282256, max=283120, avg=282722.00, stdev=306.12, samples=14
lat (usec) : 20=98.73%, 50=1.23%, 100=0.04%, 250=0.01%
cpu : usr=41.24%, sys=58.73%, ctx=20, majf=0, minf=0
IO depths : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1978816,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=966MiB (1013MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752084,
"timestamp_ms" : 1666752084789,
"time" : "Wed Oct 26 04:41:24 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1017103872,
"io_kbytes" : 993265,
"bw_bytes" : 145279798,
"bw" : 141874,
"iops" : 283749.178689,
"runtime" : 7001,
"total_ios" : 1986528,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 13124,
"mean" : 141.327949,
"stddev" : 46.350036,
"N" : 1986528
},
"clat_ns" : {
"min" : 11853,
"max" : 118873,
"mean" : 13131.754659,
"stddev" : 1847.869899,
"N" : 1986528,
"percentile" : {
"1.000000" : 12224,
"5.000000" : 12352,
"10.000000" : 12352,
"20.000000" : 12480,
"30.000000" : 12608,
"40.000000" : 12736,
"50.000000" : 12864,
"60.000000" : 12992,
"70.000000" : 12992,
"80.000000" : 13120,
"90.000000" : 13376,
"95.000000" : 13888,
"99.000000" : 22912,
"99.500000" : 23936,
"99.900000" : 29568,
"99.950000" : 35072,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 11963,
"max" : 119083,
"mean" : 13273.082608,
"stddev" : 1853.093830,
"N" : 1986528
},
"bw_min" : 141696,
"bw_max" : 142276,
"bw_agg" : 100.000000,
"bw_mean" : 141915.000000,
"bw_dev" : 201.676817,
"bw_samples" : 14,
"iops_min" : 283392,
"iops_max" : 284552,
"iops_mean" : 283829.928571,
"iops_stddev" : 403.275906,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 38.557143,
"sys_cpu" : 61.414286,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 25.000000,
"2" : 50.000000,
"4" : 25.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 98.732412,
"50" : 1.227670,
"100" : 0.039718,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=142182: Wed Oct 26 04:41:24 2022
read: IOPS=284k, BW=139MiB/s (145MB/s)(970MiB/7001msec)
slat (nsec): min=110, max=13124, avg=141.33, stdev=46.35
clat (usec): min=11, max=118, avg=13.13, stdev= 1.85
lat (usec): min=11, max=119, avg=13.27, stdev= 1.85
clat percentiles (nsec):
| 1.00th=[12224], 5.00th=[12352], 10.00th=[12352], 20.00th=[12480],
| 30.00th=[12608], 40.00th=[12736], 50.00th=[12864], 60.00th=[12992],
| 70.00th=[12992], 80.00th=[13120], 90.00th=[13376], 95.00th=[13888],
| 99.00th=[22912], 99.50th=[23936], 99.90th=[29568], 99.95th=[35072],
| 99.99th=[66048]
bw ( KiB/s): min=141696, max=142276, per=100.00%, avg=141915.00, stdev=201.68, samples=14
iops : min=283392, max=284552, avg=283829.93, stdev=403.28, samples=14
lat (usec) : 20=98.73%, 50=1.23%, 100=0.04%, 250=0.01%
cpu : usr=38.56%, sys=61.41%, ctx=23, majf=0, minf=0
IO depths : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1986528,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=139MiB/s (145MB/s), 139MiB/s-139MiB/s (145MB/s-145MB/s), io=970MiB (1017MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752099,
"timestamp_ms" : 1666752099916,
"time" : "Wed Oct 26 04:41:39 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1077817344,
"io_kbytes" : 1052556,
"bw_bytes" : 153951913,
"bw" : 150343,
"iops" : 300686.759034,
"runtime" : 7001,
"total_ios" : 2105108,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1532,
"max" : 20038,
"mean" : 2804.261997,
"stddev" : 1140.383516,
"N" : 2105109
},
"clat_ns" : {
"min" : 190,
"max" : 124213,
"mean" : 10260.727332,
"stddev" : 2255.624306,
"N" : 2105111,
"percentile" : {
"1.000000" : 8512,
"5.000000" : 8640,
"10.000000" : 8896,
"20.000000" : 9792,
"30.000000" : 9792,
"40.000000" : 9920,
"50.000000" : 9920,
"60.000000" : 9920,
"70.000000" : 10048,
"80.000000" : 10176,
"90.000000" : 11200,
"95.000000" : 11456,
"99.000000" : 20864,
"99.500000" : 23424,
"99.900000" : 29824,
"99.950000" : 38144,
"99.990000" : 68096
}
},
"lat_ns" : {
"min" : 5651,
"max" : 125786,
"mean" : 13064.989316,
"stddev" : 2336.079254,
"N" : 2105111
},
"bw_min" : 150030,
"bw_max" : 150660,
"bw_agg" : 100.000000,
"bw_mean" : 150378.142857,
"bw_dev" : 229.647508,
"bw_samples" : 14,
"iops_min" : 300060,
"iops_max" : 301320,
"iops_mean" : 300756.285714,
"iops_stddev" : 459.295016,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.271429,
"sys_cpu" : 67.700000,
"ctx" : 21,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.013824,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.021139,
"10" : 66.454738,
"20" : 32.398053,
"50" : 1.065931,
"100" : 0.044938,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142231: Wed Oct 26 04:41:39 2022
read: IOPS=301k, BW=147MiB/s (154MB/s)(1028MiB/7001msec)
slat (nsec): min=1532, max=20038, avg=2804.26, stdev=1140.38
clat (nsec): min=190, max=124213, avg=10260.73, stdev=2255.62
lat (usec): min=5, max=125, avg=13.06, stdev= 2.34
clat percentiles (nsec):
| 1.00th=[ 8512], 5.00th=[ 8640], 10.00th=[ 8896], 20.00th=[ 9792],
| 30.00th=[ 9792], 40.00th=[ 9920], 50.00th=[ 9920], 60.00th=[ 9920],
| 70.00th=[10048], 80.00th=[10176], 90.00th=[11200], 95.00th=[11456],
| 99.00th=[20864], 99.50th=[23424], 99.90th=[29824], 99.95th=[38144],
| 99.99th=[68096]
bw ( KiB/s): min=150030, max=150660, per=100.00%, avg=150378.14, stdev=229.65, samples=14
iops : min=300060, max=301320, avg=300756.29, stdev=459.30, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.02%, 10=66.45%, 20=32.40%, 50=1.07%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=32.27%, sys=67.70%, ctx=21, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2105108,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=1028MiB (1078MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752115,
"timestamp_ms" : 1666752115189,
"time" : "Wed Oct 26 04:41:55 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1060443136,
"io_kbytes" : 1035589,
"bw_bytes" : 151470237,
"bw" : 147920,
"iops" : 295839.737180,
"runtime" : 7001,
"total_ios" : 2071174,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1542,
"max" : 28865,
"mean" : 2851.573909,
"stddev" : 1251.286691,
"N" : 2071175
},
"clat_ns" : {
"min" : 190,
"max" : 120766,
"mean" : 10416.253545,
"stddev" : 2328.765258,
"N" : 2071177,
"percentile" : {
"1.000000" : 8640,
"5.000000" : 8768,
"10.000000" : 8768,
"20.000000" : 8896,
"30.000000" : 9024,
"40.000000" : 9920,
"50.000000" : 10048,
"60.000000" : 10176,
"70.000000" : 11072,
"80.000000" : 11328,
"90.000000" : 11456,
"95.000000" : 11584,
"99.000000" : 20864,
"99.500000" : 23424,
"99.900000" : 29824,
"99.950000" : 34048,
"99.990000" : 66048
}
},
"lat_ns" : {
"min" : 5650,
"max" : 122359,
"mean" : 13267.826877,
"stddev" : 2220.362076,
"N" : 2071177
},
"bw_min" : 147590,
"bw_max" : 148421,
"bw_agg" : 100.000000,
"bw_mean" : 147953.571429,
"bw_dev" : 238.447451,
"bw_samples" : 14,
"iops_min" : 295180,
"iops_max" : 296842,
"iops_mean" : 295907.071429,
"iops_stddev" : 476.984513,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.214286,
"sys_cpu" : 69.757143,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.016512,
"10" : 42.578412,
"20" : 56.246361,
"50" : 1.119945,
"100" : 0.037129,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142280: Wed Oct 26 04:41:55 2022
read: IOPS=296k, BW=144MiB/s (151MB/s)(1011MiB/7001msec)
slat (nsec): min=1542, max=28865, avg=2851.57, stdev=1251.29
clat (nsec): min=190, max=120766, avg=10416.25, stdev=2328.77
lat (usec): min=5, max=122, avg=13.27, stdev= 2.22
clat percentiles (nsec):
| 1.00th=[ 8640], 5.00th=[ 8768], 10.00th=[ 8768], 20.00th=[ 8896],
| 30.00th=[ 9024], 40.00th=[ 9920], 50.00th=[10048], 60.00th=[10176],
| 70.00th=[11072], 80.00th=[11328], 90.00th=[11456], 95.00th=[11584],
| 99.00th=[20864], 99.50th=[23424], 99.90th=[29824], 99.95th=[34048],
| 99.99th=[66048]
bw ( KiB/s): min=147590, max=148421, per=100.00%, avg=147953.57, stdev=238.45, samples=14
iops : min=295180, max=296842, avg=295907.07, stdev=476.98, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.02%, 10=42.58%, 20=56.25%, 50=1.12%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=30.21%, sys=69.76%, ctx=22, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2071174,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=1011MiB (1060MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory
$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752130,
"timestamp_ms" : 1666752130258,
"time" : "Wed Oct 26 04:42:10 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1063592960,
"io_kbytes" : 1038665,
"bw_bytes" : 151920148,
"bw" : 148359,
"iops" : 296718.468790,
"runtime" : 7001,
"total_ios" : 2077326,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1533,
"max" : 57908,
"mean" : 2828.639420,
"stddev" : 1225.631029,
"N" : 2077327
},
"clat_ns" : {
"min" : 210,
"max" : 121017,
"mean" : 10398.118086,
"stddev" : 2330.860370,
"N" : 2077329,
"percentile" : {
"1.000000" : 8640,
"5.000000" : 8768,
"10.000000" : 8768,
"20.000000" : 8896,
"30.000000" : 9920,
"40.000000" : 9920,
"50.000000" : 10048,
"60.000000" : 10176,
"70.000000" : 10944,
"80.000000" : 11200,
"90.000000" : 11328,
"95.000000" : 11584,
"99.000000" : 20864,
"99.500000" : 23424,
"99.900000" : 30080,
"99.950000" : 35072,
"99.990000" : 67072
}
},
"lat_ns" : {
"min" : 5670,
"max" : 122710,
"mean" : 13226.758125,
"stddev" : 2280.945948,
"N" : 2077329
},
"bw_min" : 147572,
"bw_max" : 148824,
"bw_agg" : 100.000000,
"bw_mean" : 148401.285714,
"bw_dev" : 298.527522,
"bw_samples" : 14,
"iops_min" : 295144,
"iops_max" : 297649,
"iops_mean" : 296802.642857,
"iops_stddev" : 597.164017,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 32.328571,
"sys_cpu" : 67.642857,
"ctx" : 29,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.023925,
"10" : 44.046240,
"20" : 54.765116,
"50" : 1.121779,
"100" : 0.039426,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142330: Wed Oct 26 04:42:10 2022
read: IOPS=297k, BW=145MiB/s (152MB/s)(1014MiB/7001msec)
slat (nsec): min=1533, max=57908, avg=2828.64, stdev=1225.63
clat (nsec): min=210, max=121017, avg=10398.12, stdev=2330.86
lat (usec): min=5, max=122, avg=13.23, stdev= 2.28
clat percentiles (nsec):
| 1.00th=[ 8640], 5.00th=[ 8768], 10.00th=[ 8768], 20.00th=[ 8896],
| 30.00th=[ 9920], 40.00th=[ 9920], 50.00th=[10048], 60.00th=[10176],
| 70.00th=[10944], 80.00th=[11200], 90.00th=[11328], 95.00th=[11584],
| 99.00th=[20864], 99.50th=[23424], 99.90th=[30080], 99.95th=[35072],
| 99.99th=[67072]
bw ( KiB/s): min=147572, max=148824, per=100.00%, avg=148401.29, stdev=298.53, samples=14
iops : min=295144, max=297649, avg=296802.64, stdev=597.16, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.02%, 10=44.05%, 20=54.77%, 50=1.12%, 100=0.04%
lat (usec) : 250=0.01%
cpu : usr=32.33%, sys=67.64%, ctx=29, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2077326,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1014MiB (1064MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752143,
"timestamp_ms" : 1666752143949,
"time" : "Wed Oct 26 04:42:23 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1086656512,
"io_kbytes" : 1061188,
"bw_bytes" : 155214471,
"bw" : 151576,
"iops" : 303152.692473,
"runtime" : 7001,
"total_ios" : 2122372,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1372,
"max" : 112582,
"mean" : 2810.550236,
"stddev" : 1247.731970,
"N" : 2122373
},
"clat_ns" : {
"min" : 110,
"max" : 125957,
"mean" : 10142.478753,
"stddev" : 2569.151273,
"N" : 2122375,
"percentile" : {
"1.000000" : 6880,
"5.000000" : 7968,
"10.000000" : 8256,
"20.000000" : 9536,
"30.000000" : 9664,
"40.000000" : 9664,
"50.000000" : 9792,
"60.000000" : 9792,
"70.000000" : 9792,
"80.000000" : 10048,
"90.000000" : 11456,
"95.000000" : 11968,
"99.000000" : 23424,
"99.500000" : 24960,
"99.900000" : 29824,
"99.950000" : 34048,
"99.990000" : 70144
}
},
"lat_ns" : {
"min" : 5921,
"max" : 127369,
"mean" : 12953.029064,
"stddev" : 2798.630057,
"N" : 2122375
},
"bw_min" : 151380,
"bw_max" : 151845,
"bw_agg" : 100.000000,
"bw_mean" : 151616.571429,
"bw_dev" : 148.988236,
"bw_samples" : 14,
"iops_min" : 302760,
"iops_max" : 303691,
"iops_mean" : 303233.214286,
"iops_stddev" : 298.105861,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 18.328571,
"sys_cpu" : 79.457143,
"ctx" : 12395,
"majf" : 0,
"minf" : 1,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.024925,
"4" : 0.110160,
"10" : 80.312170,
"20" : 17.752307,
"50" : 1.758127,
"100" : 0.039578,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=142369: Wed Oct 26 04:42:23 2022
read: IOPS=303k, BW=148MiB/s (155MB/s)(1036MiB/7001msec)
slat (nsec): min=1372, max=112582, avg=2810.55, stdev=1247.73
clat (nsec): min=110, max=125957, avg=10142.48, stdev=2569.15
lat (usec): min=5, max=127, avg=12.95, stdev= 2.80
clat percentiles (nsec):
| 1.00th=[ 6880], 5.00th=[ 7968], 10.00th=[ 8256], 20.00th=[ 9536],
| 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[ 9792], 60.00th=[ 9792],
| 70.00th=[ 9792], 80.00th=[10048], 90.00th=[11456], 95.00th=[11968],
| 99.00th=[23424], 99.50th=[24960], 99.90th=[29824], 99.95th=[34048],
| 99.99th=[70144]
bw ( KiB/s): min=151380, max=151845, per=100.00%, avg=151616.57, stdev=148.99, samples=14
iops : min=302760, max=303691, avg=303233.21, stdev=298.11, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.02%, 4=0.11%, 10=80.31%, 20=17.75%, 50=1.76%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=18.33%, sys=79.46%, ctx=12395, majf=0, minf=1
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2122372,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1036MiB (1087MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752157,
"timestamp_ms" : 1666752157934,
"time" : "Wed Oct 26 04:42:37 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1083688448,
"io_kbytes" : 1058289,
"bw_bytes" : 154790522,
"bw" : 151162,
"iops" : 302324.667905,
"runtime" : 7001,
"total_ios" : 2116575,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1362,
"max" : 21871,
"mean" : 2810.793388,
"stddev" : 1238.113474,
"N" : 2116576
},
"clat_ns" : {
"min" : 110,
"max" : 126127,
"mean" : 10178.077711,
"stddev" : 2575.128921,
"N" : 2116578,
"percentile" : {
"1.000000" : 6880,
"5.000000" : 7968,
"10.000000" : 8512,
"20.000000" : 9536,
"30.000000" : 9664,
"40.000000" : 9664,
"50.000000" : 9792,
"60.000000" : 9792,
"70.000000" : 9920,
"80.000000" : 10048,
"90.000000" : 11456,
"95.000000" : 11968,
"99.000000" : 23424,
"99.500000" : 24960,
"99.900000" : 30336,
"99.950000" : 34560,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 5841,
"max" : 127520,
"mean" : 12988.870477,
"stddev" : 2788.918764,
"N" : 2116578
},
"bw_min" : 150921,
"bw_max" : 151477,
"bw_agg" : 100.000000,
"bw_mean" : 151202.642857,
"bw_dev" : 172.050801,
"bw_samples" : 14,
"iops_min" : 301842,
"iops_max" : 302954,
"iops_mean" : 302405.500000,
"iops_stddev" : 344.057408,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.585714,
"sys_cpu" : 80.171429,
"ctx" : 12447,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.019702,
"4" : 0.111123,
"10" : 79.619905,
"20" : 18.496203,
"50" : 1.703979,
"100" : 0.044884,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=142408: Wed Oct 26 04:42:37 2022
read: IOPS=302k, BW=148MiB/s (155MB/s)(1033MiB/7001msec)
slat (nsec): min=1362, max=21871, avg=2810.79, stdev=1238.11
clat (nsec): min=110, max=126127, avg=10178.08, stdev=2575.13
lat (usec): min=5, max=127, avg=12.99, stdev= 2.79
clat percentiles (nsec):
| 1.00th=[ 6880], 5.00th=[ 7968], 10.00th=[ 8512], 20.00th=[ 9536],
| 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[ 9792], 60.00th=[ 9792],
| 70.00th=[ 9920], 80.00th=[10048], 90.00th=[11456], 95.00th=[11968],
| 99.00th=[23424], 99.50th=[24960], 99.90th=[30336], 99.95th=[34560],
| 99.99th=[71168]
bw ( KiB/s): min=150921, max=151477, per=100.00%, avg=151202.64, stdev=172.05, samples=14
iops : min=301842, max=302954, avg=302405.50, stdev=344.06, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.02%, 4=0.11%, 10=79.62%, 20=18.50%, 50=1.70%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=17.59%, sys=80.17%, ctx=12447, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2116575,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1033MiB (1084MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_io_uring-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752171,
"timestamp_ms" : 1666752171569,
"time" : "Wed Oct 26 04:42:51 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1086826496,
"io_kbytes" : 1061354,
"bw_bytes" : 155238751,
"bw" : 151600,
"iops" : 303200.114269,
"runtime" : 7001,
"total_ios" : 2122704,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1362,
"max" : 69741,
"mean" : 2801.643828,
"stddev" : 1238.790601,
"N" : 2122705
},
"clat_ns" : {
"min" : 110,
"max" : 124213,
"mean" : 10150.495425,
"stddev" : 2562.103546,
"N" : 2122707,
"percentile" : {
"1.000000" : 6880,
"5.000000" : 7968,
"10.000000" : 9408,
"20.000000" : 9536,
"30.000000" : 9664,
"40.000000" : 9664,
"50.000000" : 9792,
"60.000000" : 9792,
"70.000000" : 9920,
"80.000000" : 9920,
"90.000000" : 11328,
"95.000000" : 11840,
"99.000000" : 23424,
"99.500000" : 24960,
"99.900000" : 30336,
"99.950000" : 34560,
"99.990000" : 70144
}
},
"lat_ns" : {
"min" : 5861,
"max" : 125596,
"mean" : 12952.138615,
"stddev" : 2763.745120,
"N" : 2122707
},
"bw_min" : 151397,
"bw_max" : 151887,
"bw_agg" : 100.000000,
"bw_mean" : 151658.000000,
"bw_dev" : 143.826864,
"bw_samples" : 13,
"iops_min" : 302794,
"iops_max" : 303775,
"iops_mean" : 303316.076923,
"iops_stddev" : 287.786513,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.157143,
"sys_cpu" : 78.600000,
"ctx" : 12468,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.049371,
"4" : 0.088990,
"10" : 81.786580,
"20" : 16.352115,
"50" : 1.670982,
"100" : 0.044519,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=142447: Wed Oct 26 04:42:51 2022
read: IOPS=303k, BW=148MiB/s (155MB/s)(1036MiB/7001msec)
slat (nsec): min=1362, max=69741, avg=2801.64, stdev=1238.79
clat (nsec): min=110, max=124213, avg=10150.50, stdev=2562.10
lat (usec): min=5, max=125, avg=12.95, stdev= 2.76
clat percentiles (nsec):
| 1.00th=[ 6880], 5.00th=[ 7968], 10.00th=[ 9408], 20.00th=[ 9536],
| 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[ 9792], 60.00th=[ 9792],
| 70.00th=[ 9920], 80.00th=[ 9920], 90.00th=[11328], 95.00th=[11840],
| 99.00th=[23424], 99.50th=[24960], 99.90th=[30336], 99.95th=[34560],
| 99.99th=[70144]
bw ( KiB/s): min=151397, max=151887, per=100.00%, avg=151658.00, stdev=143.83, samples=13
iops : min=302794, max=303775, avg=303316.08, stdev=287.79, samples=13
lat (nsec) : 250=0.01%
lat (usec) : 2=0.05%, 4=0.09%, 10=81.79%, 20=16.35%, 50=1.67%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=19.16%, sys=78.60%, ctx=12468, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2122704,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1036MiB (1087MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-reference_0.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752185,
"timestamp_ms" : 1666752185221,
"time" : "Wed Oct 26 04:43:05 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1041431040,
"io_kbytes" : 1017022,
"bw_bytes" : 148754612,
"bw" : 145268,
"iops" : 290535.780603,
"runtime" : 7001,
"total_ios" : 2034041,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1403,
"max" : 42539,
"mean" : 2718.299399,
"stddev" : 1303.395648,
"N" : 2034042
},
"clat_ns" : {
"min" : 4479,
"max" : 124985,
"mean" : 10812.651804,
"stddev" : 2728.741015,
"N" : 2034044,
"percentile" : {
"1.000000" : 9024,
"5.000000" : 9024,
"10.000000" : 9024,
"20.000000" : 9152,
"30.000000" : 9280,
"40.000000" : 10304,
"50.000000" : 10432,
"60.000000" : 10560,
"70.000000" : 11584,
"80.000000" : 11712,
"90.000000" : 11840,
"95.000000" : 12096,
"99.000000" : 23680,
"99.500000" : 28800,
"99.900000" : 35072,
"99.950000" : 40704,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 8055,
"max" : 129283,
"mean" : 13530.950614,
"stddev" : 2627.157009,
"N" : 2034044
},
"bw_min" : 144981,
"bw_max" : 145561,
"bw_agg" : 100.000000,
"bw_mean" : 145329.000000,
"bw_dev" : 165.697593,
"bw_samples" : 14,
"iops_min" : 289962,
"iops_max" : 291122,
"iops_mean" : 290658.071429,
"iops_stddev" : 331.438465,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 34.800000,
"sys_cpu" : 63.071429,
"ctx" : 12160,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 32.978195,
"20" : 65.542287,
"50" : 1.435320,
"100" : 0.043804,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2916402,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 15253,
"write_ticks" : 0,
"in_queue" : 15253,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=142487: Wed Oct 26 04:43:05 2022
read: IOPS=291k, BW=142MiB/s (149MB/s)(993MiB/7001msec)
slat (nsec): min=1403, max=42539, avg=2718.30, stdev=1303.40
clat (usec): min=4, max=124, avg=10.81, stdev= 2.73
lat (usec): min=8, max=129, avg=13.53, stdev= 2.63
clat percentiles (nsec):
| 1.00th=[ 9024], 5.00th=[ 9024], 10.00th=[ 9024], 20.00th=[ 9152],
| 30.00th=[ 9280], 40.00th=[10304], 50.00th=[10432], 60.00th=[10560],
| 70.00th=[11584], 80.00th=[11712], 90.00th=[11840], 95.00th=[12096],
| 99.00th=[23680], 99.50th=[28800], 99.90th=[35072], 99.95th=[40704],
| 99.99th=[72192]
bw ( KiB/s): min=144981, max=145561, per=100.00%, avg=145329.00, stdev=165.70, samples=14
iops : min=289962, max=291122, avg=290658.07, stdev=331.44, samples=14
lat (usec) : 10=32.98%, 20=65.54%, 50=1.44%, 100=0.04%, 250=0.01%
cpu : usr=34.80%, sys=63.07%, ctx=12160, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2034041,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=993MiB (1041MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2916402/0, merge=0/0, ticks=15253/0, in_queue=15253, util=99.09%
$ rm /tmp/fio-output_512_4_libaio-reference_1.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752198,
"timestamp_ms" : 1666752198867,
"time" : "Wed Oct 26 04:43:18 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1038159872,
"io_kbytes" : 1013828,
"bw_bytes" : 148287369,
"bw" : 144811,
"iops" : 289623.196686,
"runtime" : 7001,
"total_ios" : 2027652,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1402,
"max" : 25027,
"mean" : 2728.420388,
"stddev" : 1317.432382,
"N" : 2027653
},
"clat_ns" : {
"min" : 4689,
"max" : 93515,
"mean" : 10845.511402,
"stddev" : 2729.393787,
"N" : 2027655,
"percentile" : {
"1.000000" : 9024,
"5.000000" : 9024,
"10.000000" : 9152,
"20.000000" : 9152,
"30.000000" : 9280,
"40.000000" : 10304,
"50.000000" : 10432,
"60.000000" : 10816,
"70.000000" : 11584,
"80.000000" : 11712,
"90.000000" : 11840,
"95.000000" : 11968,
"99.000000" : 23424,
"99.500000" : 29056,
"99.900000" : 35072,
"99.950000" : 40192,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 8115,
"max" : 98064,
"mean" : 13573.932355,
"stddev" : 2591.494403,
"N" : 2027655
},
"bw_min" : 144516,
"bw_max" : 145113,
"bw_agg" : 100.000000,
"bw_mean" : 144877.230769,
"bw_dev" : 161.265906,
"bw_samples" : 13,
"iops_min" : 289032,
"iops_max" : 290226,
"iops_mean" : 289754.692308,
"iops_stddev" : 322.712820,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 34.757143,
"sys_cpu" : 63.128571,
"ctx" : 12125,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 36.527964,
"20" : 61.978979,
"50" : 1.450989,
"100" : 0.042216,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2908270,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 15201,
"write_ticks" : 0,
"in_queue" : 15201,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=142526: Wed Oct 26 04:43:18 2022
read: IOPS=290k, BW=141MiB/s (148MB/s)(990MiB/7001msec)
slat (nsec): min=1402, max=25027, avg=2728.42, stdev=1317.43
clat (nsec): min=4689, max=93515, avg=10845.51, stdev=2729.39
lat (nsec): min=8115, max=98064, avg=13573.93, stdev=2591.49
clat percentiles (nsec):
| 1.00th=[ 9024], 5.00th=[ 9024], 10.00th=[ 9152], 20.00th=[ 9152],
| 30.00th=[ 9280], 40.00th=[10304], 50.00th=[10432], 60.00th=[10816],
| 70.00th=[11584], 80.00th=[11712], 90.00th=[11840], 95.00th=[11968],
| 99.00th=[23424], 99.50th=[29056], 99.90th=[35072], 99.95th=[40192],
| 99.99th=[71168]
bw ( KiB/s): min=144516, max=145113, per=100.00%, avg=144877.23, stdev=161.27, samples=13
iops : min=289032, max=290226, avg=289754.69, stdev=322.71, samples=13
lat (usec) : 10=36.53%, 20=61.98%, 50=1.45%, 100=0.04%
cpu : usr=34.76%, sys=63.13%, ctx=12125, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2027652,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=990MiB (1038MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2908270/0, merge=0/0, ticks=15201/0, in_queue=15201, util=99.09%
$ rm /tmp/fio-output_512_4_libaio-reference_2.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751053,
"timestamp_ms" : 1666751053526,
"time" : "Wed Oct 26 04:24:13 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 439462400,
"io_kbytes" : 429162,
"bw_bytes" : 62771375,
"bw" : 61300,
"iops" : 122600.199971,
"runtime" : 7001,
"total_ios" : 858324,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 701,
"max" : 13997,
"mean" : 918.099893,
"stddev" : 66.379994,
"N" : 858325
},
"clat_ns" : {
"min" : 551,
"max" : 101871,
"mean" : 6972.345603,
"stddev" : 1116.060535,
"N" : 858324,
"percentile" : {
"1.000000" : 6368,
"5.000000" : 6496,
"10.000000" : 6560,
"20.000000" : 6688,
"30.000000" : 6752,
"40.000000" : 6752,
"50.000000" : 6816,
"60.000000" : 6944,
"70.000000" : 6944,
"80.000000" : 7008,
"90.000000" : 7136,
"95.000000" : 7264,
"99.000000" : 12864,
"99.500000" : 13632,
"99.900000" : 16512,
"99.950000" : 16768,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 7013,
"max" : 102792,
"mean" : 7890.445503,
"stddev" : 1119.494292,
"N" : 858324
},
"bw_min" : 61255,
"bw_max" : 61417,
"bw_agg" : 100.000000,
"bw_mean" : 61317.071429,
"bw_dev" : 43.492642,
"bw_samples" : 14,
"iops_min" : 122510,
"iops_max" : 122834,
"iops_mean" : 122634.214286,
"iops_stddev" : 86.918334,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.301574,
"20" : 1.679669,
"50" : 0.010000,
"100" : 0.014563,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139019: Wed Oct 26 04:24:13 2022
read: IOPS=123k, BW=59.9MiB/s (62.8MB/s)(419MiB/7001msec)
slat (nsec): min=701, max=13997, avg=918.10, stdev=66.38
clat (nsec): min=551, max=101871, avg=6972.35, stdev=1116.06
lat (usec): min=7, max=102, avg= 7.89, stdev= 1.12
clat percentiles (nsec):
| 1.00th=[ 6368], 5.00th=[ 6496], 10.00th=[ 6560], 20.00th=[ 6688],
| 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6816], 60.00th=[ 6944],
| 70.00th=[ 6944], 80.00th=[ 7008], 90.00th=[ 7136], 95.00th=[ 7264],
| 99.00th=[12864], 99.50th=[13632], 99.90th=[16512], 99.95th=[16768],
| 99.99th=[58112]
bw ( KiB/s): min=61255, max=61417, per=100.00%, avg=61317.07, stdev=43.49, samples=14
iops : min=122510, max=122834, avg=122634.21, stdev=86.92, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 10=98.30%, 20=1.68%, 50=0.01%, 100=0.01%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=22, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=858324,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=59.9MiB/s (62.8MB/s), 59.9MiB/s-59.9MiB/s (62.8MB/s-62.8MB/s), io=419MiB (439MB), run=7001-7001msec
$ cat /tmp/fio-output_512_4_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752212,
"timestamp_ms" : 1666752212538,
"time" : "Wed Oct 26 04:43:32 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1037223424,
"io_kbytes" : 1012913,
"bw_bytes" : 148153610,
"bw" : 144681,
"iops" : 289361.948293,
"runtime" : 7001,
"total_ios" : 2025823,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1412,
"max" : 28864,
"mean" : 2730.817650,
"stddev" : 1328.449286,
"N" : 2025824
},
"clat_ns" : {
"min" : 4679,
"max" : 134222,
"mean" : 10856.180382,
"stddev" : 2794.572731,
"N" : 2025826,
"percentile" : {
"1.000000" : 9024,
"5.000000" : 9024,
"10.000000" : 9152,
"20.000000" : 9152,
"30.000000" : 9280,
"40.000000" : 10304,
"50.000000" : 10432,
"60.000000" : 10816,
"70.000000" : 11584,
"80.000000" : 11712,
"90.000000" : 11840,
"95.000000" : 12096,
"99.000000" : 23680,
"99.500000" : 29056,
"99.900000" : 35584,
"99.950000" : 42240,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 8135,
"max" : 137117,
"mean" : 13586.998051,
"stddev" : 2658.191293,
"N" : 2025826
},
"bw_min" : 144341,
"bw_max" : 145442,
"bw_agg" : 100.000000,
"bw_mean" : 144773.769231,
"bw_dev" : 340.786383,
"bw_samples" : 13,
"iops_min" : 288682,
"iops_max" : 290884,
"iops_mean" : 289547.461538,
"iops_stddev" : 681.758341,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 35.828571,
"sys_cpu" : 62.028571,
"ctx" : 12148,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 36.357915,
"20" : 62.091604,
"50" : 1.503537,
"100" : 0.046253,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2832626,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 14860,
"write_ticks" : 0,
"in_queue" : 14860,
"util" : 99.085923
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=142565: Wed Oct 26 04:43:32 2022
read: IOPS=289k, BW=141MiB/s (148MB/s)(989MiB/7001msec)
slat (nsec): min=1412, max=28864, avg=2730.82, stdev=1328.45
clat (usec): min=4, max=134, avg=10.86, stdev= 2.79
lat (usec): min=8, max=137, avg=13.59, stdev= 2.66
clat percentiles (nsec):
| 1.00th=[ 9024], 5.00th=[ 9024], 10.00th=[ 9152], 20.00th=[ 9152],
| 30.00th=[ 9280], 40.00th=[10304], 50.00th=[10432], 60.00th=[10816],
| 70.00th=[11584], 80.00th=[11712], 90.00th=[11840], 95.00th=[12096],
| 99.00th=[23680], 99.50th=[29056], 99.90th=[35584], 99.95th=[42240],
| 99.99th=[72192]
bw ( KiB/s): min=144341, max=145442, per=100.00%, avg=144773.77, stdev=340.79, samples=13
iops : min=288682, max=290884, avg=289547.46, stdev=681.76, samples=13
lat (usec) : 10=36.36%, 20=62.09%, 50=1.50%, 100=0.05%, 250=0.01%
cpu : usr=35.83%, sys=62.03%, ctx=12148, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2025823,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=989MiB (1037MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2832626/0, merge=0/0, ticks=14860/0, in_queue=14860, util=99.09%
$ rm /tmp/fio-output_512_4_libaio-bdev_aio_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752227,
"timestamp_ms" : 1666752227603,
"time" : "Wed Oct 26 04:43:47 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1071235584,
"io_kbytes" : 1046128,
"bw_bytes" : 153011796,
"bw" : 149425,
"iops" : 298850.878446,
"runtime" : 7001,
"total_ios" : 2092255,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1583,
"max" : 29135,
"mean" : 2702.866753,
"stddev" : 1007.331736,
"N" : 2092256
},
"clat_ns" : {
"min" : 260,
"max" : 114535,
"mean" : 7822.386945,
"stddev" : 3118.577266,
"N" : 2092257,
"percentile" : {
"1.000000" : 3280,
"5.000000" : 3376,
"10.000000" : 4512,
"20.000000" : 4832,
"30.000000" : 6432,
"40.000000" : 6688,
"50.000000" : 7200,
"60.000000" : 8512,
"70.000000" : 9792,
"80.000000" : 10048,
"90.000000" : 11072,
"95.000000" : 11456,
"99.000000" : 18560,
"99.500000" : 21376,
"99.900000" : 24960,
"99.950000" : 31872,
"99.990000" : 63744
}
},
"lat_ns" : {
"min" : 5791,
"max" : 116228,
"mean" : 10525.253804,
"stddev" : 3235.371221,
"N" : 2092257
},
"bw_min" : 149025,
"bw_max" : 149805,
"bw_agg" : 100.000000,
"bw_mean" : 149469.785714,
"bw_dev" : 290.944134,
"bw_samples" : 14,
"iops_min" : 298050,
"iops_max" : 299606,
"iops_mean" : 298939.571429,
"iops_stddev" : 581.128440,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 43.357143,
"sys_cpu" : 56.614286,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.967043,
"2" : 60.186784,
"4" : 38.846173,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.128904,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.015390,
"4" : 7.186170,
"10" : 73.121154,
"20" : 18.881733,
"50" : 0.629417,
"100" : 0.036946,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=142613: Wed Oct 26 04:43:47 2022
read: IOPS=299k, BW=146MiB/s (153MB/s)(1022MiB/7001msec)
slat (nsec): min=1583, max=29135, avg=2702.87, stdev=1007.33
clat (nsec): min=260, max=114535, avg=7822.39, stdev=3118.58
lat (usec): min=5, max=116, avg=10.53, stdev= 3.24
clat percentiles (nsec):
| 1.00th=[ 3280], 5.00th=[ 3376], 10.00th=[ 4512], 20.00th=[ 4832],
| 30.00th=[ 6432], 40.00th=[ 6688], 50.00th=[ 7200], 60.00th=[ 8512],
| 70.00th=[ 9792], 80.00th=[10048], 90.00th=[11072], 95.00th=[11456],
| 99.00th=[18560], 99.50th=[21376], 99.90th=[24960], 99.95th=[31872],
| 99.99th=[63744]
bw ( KiB/s): min=149025, max=149805, per=100.00%, avg=149469.79, stdev=290.94, samples=14
iops : min=298050, max=299606, avg=298939.57, stdev=581.13, samples=14
lat (nsec) : 500=0.13%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.02%, 4=7.19%, 10=73.12%, 20=18.88%, 50=0.63%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=43.36%, sys=56.61%, ctx=22, majf=0, minf=0
IO depths : 1=1.0%, 2=60.2%, 4=38.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2092255,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1022MiB (1071MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-bdev_aio_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752242,
"timestamp_ms" : 1666752242721,
"time" : "Wed Oct 26 04:44:02 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1061995520,
"io_kbytes" : 1037105,
"bw_bytes" : 151691975,
"bw" : 148136,
"iops" : 296273.103842,
"runtime" : 7001,
"total_ios" : 2074208,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1603,
"max" : 57037,
"mean" : 2788.654794,
"stddev" : 1082.503450,
"N" : 2074209
},
"clat_ns" : {
"min" : 241,
"max" : 157215,
"mean" : 7732.501443,
"stddev" : 3123.606252,
"N" : 2074210,
"percentile" : {
"1.000000" : 3312,
"5.000000" : 4512,
"10.000000" : 4576,
"20.000000" : 4704,
"30.000000" : 6432,
"40.000000" : 6560,
"50.000000" : 6880,
"60.000000" : 8256,
"70.000000" : 9920,
"80.000000" : 10048,
"90.000000" : 11200,
"95.000000" : 11456,
"99.000000" : 18304,
"99.500000" : 21120,
"99.900000" : 25216,
"99.950000" : 31872,
"99.990000" : 62720
}
},
"lat_ns" : {
"min" : 5721,
"max" : 158908,
"mean" : 10521.156284,
"stddev" : 3475.499276,
"N" : 2074210
},
"bw_min" : 147810,
"bw_max" : 148550,
"bw_agg" : 100.000000,
"bw_mean" : 148173.714286,
"bw_dev" : 244.282443,
"bw_samples" : 14,
"iops_min" : 295620,
"iops_max" : 297100,
"iops_mean" : 296347.428571,
"iops_stddev" : 488.564887,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 43.757143,
"sys_cpu" : 56.214286,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.941564,
"2" : 61.963410,
"4" : 37.095026,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.128868,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.017645,
"4" : 4.562320,
"10" : 70.359482,
"20" : 24.313714,
"50" : 0.580125,
"100" : 0.036978,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=142662: Wed Oct 26 04:44:02 2022
read: IOPS=296k, BW=145MiB/s (152MB/s)(1013MiB/7001msec)
slat (nsec): min=1603, max=57037, avg=2788.65, stdev=1082.50
clat (nsec): min=241, max=157215, avg=7732.50, stdev=3123.61
lat (usec): min=5, max=158, avg=10.52, stdev= 3.48
clat percentiles (nsec):
| 1.00th=[ 3312], 5.00th=[ 4512], 10.00th=[ 4576], 20.00th=[ 4704],
| 30.00th=[ 6432], 40.00th=[ 6560], 50.00th=[ 6880], 60.00th=[ 8256],
| 70.00th=[ 9920], 80.00th=[10048], 90.00th=[11200], 95.00th=[11456],
| 99.00th=[18304], 99.50th=[21120], 99.90th=[25216], 99.95th=[31872],
| 99.99th=[62720]
bw ( KiB/s): min=147810, max=148550, per=100.00%, avg=148173.71, stdev=244.28, samples=14
iops : min=295620, max=297100, avg=296347.43, stdev=488.56, samples=14
lat (nsec) : 250=0.01%, 500=0.13%, 750=0.01%
lat (usec) : 2=0.02%, 4=4.56%, 10=70.36%, 20=24.31%, 50=0.58%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=43.76%, sys=56.21%, ctx=23, majf=0, minf=0
IO depths : 1=0.9%, 2=62.0%, 4=37.1%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2074208,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1013MiB (1062MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-bdev_aio_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752257,
"timestamp_ms" : 1666752257808,
"time" : "Wed Oct 26 04:44:17 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1064985600,
"io_kbytes" : 1040025,
"bw_bytes" : 152119068,
"bw" : 148553,
"iops" : 297107.270390,
"runtime" : 7001,
"total_ios" : 2080048,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1593,
"max" : 32571,
"mean" : 2739.548837,
"stddev" : 1047.822313,
"N" : 2080049
},
"clat_ns" : {
"min" : 270,
"max" : 117951,
"mean" : 7801.096301,
"stddev" : 3165.549384,
"N" : 2080050,
"percentile" : {
"1.000000" : 3312,
"5.000000" : 3440,
"10.000000" : 4576,
"20.000000" : 4704,
"30.000000" : 6432,
"40.000000" : 6560,
"50.000000" : 6944,
"60.000000" : 8512,
"70.000000" : 9920,
"80.000000" : 10048,
"90.000000" : 11200,
"95.000000" : 11456,
"99.000000" : 18304,
"99.500000" : 21120,
"99.900000" : 25472,
"99.950000" : 32128,
"99.990000" : 64256
}
},
"lat_ns" : {
"min" : 5830,
"max" : 119664,
"mean" : 10540.645231,
"stddev" : 3389.356523,
"N" : 2080050
},
"bw_min" : 148084,
"bw_max" : 149034,
"bw_agg" : 100.000000,
"bw_mean" : 148592.357143,
"bw_dev" : 285.577043,
"bw_samples" : 14,
"iops_min" : 296168,
"iops_max" : 298068,
"iops_mean" : 297184.714286,
"iops_stddev" : 571.154085,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 41.985714,
"sys_cpu" : 57.985714,
"ctx" : 25,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.959641,
"2" : 61.257433,
"4" : 37.782926,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.123459,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.015096,
"4" : 6.310912,
"10" : 69.856369,
"20" : 23.058602,
"50" : 0.596188,
"100" : 0.038797,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=142712: Wed Oct 26 04:44:17 2022
read: IOPS=297k, BW=145MiB/s (152MB/s)(1016MiB/7001msec)
slat (nsec): min=1593, max=32571, avg=2739.55, stdev=1047.82
clat (nsec): min=270, max=117951, avg=7801.10, stdev=3165.55
lat (usec): min=5, max=119, avg=10.54, stdev= 3.39
clat percentiles (nsec):
| 1.00th=[ 3312], 5.00th=[ 3440], 10.00th=[ 4576], 20.00th=[ 4704],
| 30.00th=[ 6432], 40.00th=[ 6560], 50.00th=[ 6944], 60.00th=[ 8512],
| 70.00th=[ 9920], 80.00th=[10048], 90.00th=[11200], 95.00th=[11456],
| 99.00th=[18304], 99.50th=[21120], 99.90th=[25472], 99.95th=[32128],
| 99.99th=[64256]
bw ( KiB/s): min=148084, max=149034, per=100.00%, avg=148592.36, stdev=285.58, samples=14
iops : min=296168, max=298068, avg=297184.71, stdev=571.15, samples=14
lat (nsec) : 500=0.12%
lat (usec) : 2=0.02%, 4=6.31%, 10=69.86%, 20=23.06%, 50=0.60%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=41.99%, sys=57.99%, ctx=25, majf=0, minf=0
IO depths : 1=1.0%, 2=61.3%, 4=37.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2080048,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1016MiB (1065MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-reference_0.txt
$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752273,
"timestamp_ms" : 1666752273038,
"time" : "Wed Oct 26 04:44:33 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1012254720,
"io_kbytes" : 988530,
"bw_bytes" : 144587161,
"bw" : 141198,
"iops" : 282396.514784,
"runtime" : 7001,
"total_ios" : 1977058,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1603,
"max" : 63850,
"mean" : 2849.242207,
"stddev" : 1275.330036,
"N" : 1977059
},
"clat_ns" : {
"min" : 641,
"max" : 100899,
"mean" : 8513.515201,
"stddev" : 2759.933150,
"N" : 1977060,
"percentile" : {
"1.000000" : 4832,
"5.000000" : 4960,
"10.000000" : 5088,
"20.000000" : 6880,
"30.000000" : 7456,
"40.000000" : 7584,
"50.000000" : 7648,
"60.000000" : 9280,
"70.000000" : 9408,
"80.000000" : 9536,
"90.000000" : 11328,
"95.000000" : 12224,
"99.000000" : 18560,
"99.500000" : 21120,
"99.900000" : 25216,
"99.950000" : 31872,
"99.990000" : 63744
}
},
"lat_ns" : {
"min" : 6052,
"max" : 102622,
"mean" : 11362.757426,
"stddev" : 3379.702263,
"N" : 1977060
},
"bw_min" : 140715,
"bw_max" : 141758,
"bw_agg" : 100.000000,
"bw_mean" : 141236.428571,
"bw_dev" : 312.103216,
"bw_samples" : 14,
"iops_min" : 281430,
"iops_max" : 283516,
"iops_mean" : 282472.928571,
"iops_stddev" : 624.256415,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 43.642857,
"sys_cpu" : 56.328571,
"ctx" : 24,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.877010,
"2" : 56.256974,
"4" : 42.866016,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.053918,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.015680,
"4" : 0.791783,
"10" : 82.251962,
"20" : 16.201093,
"50" : 0.643279,
"100" : 0.037986,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142761: Wed Oct 26 04:44:33 2022
read: IOPS=282k, BW=138MiB/s (145MB/s)(965MiB/7001msec)
slat (nsec): min=1603, max=63850, avg=2849.24, stdev=1275.33
clat (nsec): min=641, max=100899, avg=8513.52, stdev=2759.93
lat (usec): min=6, max=102, avg=11.36, stdev= 3.38
clat percentiles (nsec):
| 1.00th=[ 4832], 5.00th=[ 4960], 10.00th=[ 5088], 20.00th=[ 6880],
| 30.00th=[ 7456], 40.00th=[ 7584], 50.00th=[ 7648], 60.00th=[ 9280],
| 70.00th=[ 9408], 80.00th=[ 9536], 90.00th=[11328], 95.00th=[12224],
| 99.00th=[18560], 99.50th=[21120], 99.90th=[25216], 99.95th=[31872],
| 99.99th=[63744]
bw ( KiB/s): min=140715, max=141758, per=100.00%, avg=141236.43, stdev=312.10, samples=14
iops : min=281430, max=283516, avg=282472.93, stdev=624.26, samples=14
lat (nsec) : 750=0.05%, 1000=0.01%
lat (usec) : 2=0.02%, 4=0.79%, 10=82.25%, 20=16.20%, 50=0.64%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=43.64%, sys=56.33%, ctx=24, majf=0, minf=0
IO depths : 1=0.9%, 2=56.3%, 4=42.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1977058,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=965MiB (1012MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752288,
"timestamp_ms" : 1666752288178,
"time" : "Wed Oct 26 04:44:48 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1009187328,
"io_kbytes" : 985534,
"bw_bytes" : 144149025,
"bw" : 140770,
"iops" : 281540.494215,
"runtime" : 7001,
"total_ios" : 1971065,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1613,
"max" : 36499,
"mean" : 2864.161118,
"stddev" : 1277.898171,
"N" : 1971066
},
"clat_ns" : {
"min" : 641,
"max" : 123301,
"mean" : 8534.661721,
"stddev" : 2698.421230,
"N" : 1971065,
"percentile" : {
"1.000000" : 4896,
"5.000000" : 5024,
"10.000000" : 5152,
"20.000000" : 6944,
"30.000000" : 7456,
"40.000000" : 7584,
"50.000000" : 7712,
"60.000000" : 9280,
"70.000000" : 9408,
"80.000000" : 9536,
"90.000000" : 10944,
"95.000000" : 12224,
"99.000000" : 18304,
"99.500000" : 21120,
"99.900000" : 25472,
"99.950000" : 32384,
"99.990000" : 63744
}
},
"lat_ns" : {
"min" : 6042,
"max" : 127870,
"mean" : 11398.823449,
"stddev" : 3337.911286,
"N" : 1971065
},
"bw_min" : 139935,
"bw_max" : 141743,
"bw_agg" : 100.000000,
"bw_mean" : 140813.428571,
"bw_dev" : 451.996207,
"bw_samples" : 14,
"iops_min" : 279870,
"iops_max" : 283486,
"iops_mean" : 281626.928571,
"iops_stddev" : 904.069642,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 44.785714,
"sys_cpu" : 55.185714,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.869327,
"2" : 55.711506,
"4" : 43.419167,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.051191,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.016691,
"4" : 0.692570,
"10" : 83.599323,
"20" : 14.998186,
"50" : 0.595871,
"100" : 0.041500,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142811: Wed Oct 26 04:44:48 2022
read: IOPS=282k, BW=137MiB/s (144MB/s)(962MiB/7001msec)
slat (nsec): min=1613, max=36499, avg=2864.16, stdev=1277.90
clat (nsec): min=641, max=123301, avg=8534.66, stdev=2698.42
lat (usec): min=6, max=127, avg=11.40, stdev= 3.34
clat percentiles (nsec):
| 1.00th=[ 4896], 5.00th=[ 5024], 10.00th=[ 5152], 20.00th=[ 6944],
| 30.00th=[ 7456], 40.00th=[ 7584], 50.00th=[ 7712], 60.00th=[ 9280],
| 70.00th=[ 9408], 80.00th=[ 9536], 90.00th=[10944], 95.00th=[12224],
| 99.00th=[18304], 99.50th=[21120], 99.90th=[25472], 99.95th=[32384],
| 99.99th=[63744]
bw ( KiB/s): min=139935, max=141743, per=100.00%, avg=140813.43, stdev=452.00, samples=14
iops : min=279870, max=283486, avg=281626.93, stdev=904.07, samples=14
lat (nsec) : 750=0.05%, 1000=0.01%
lat (usec) : 2=0.02%, 4=0.69%, 10=83.60%, 20=15.00%, 50=0.60%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=44.79%, sys=55.19%, ctx=22, majf=0, minf=0
IO depths : 1=0.9%, 2=55.7%, 4=43.4%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1971065,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=962MiB (1009MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752303,
"timestamp_ms" : 1666752303292,
"time" : "Wed Oct 26 04:45:03 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1011238912,
"io_kbytes" : 987538,
"bw_bytes" : 144442067,
"bw" : 141056,
"iops" : 282112.841023,
"runtime" : 7001,
"total_ios" : 1975072,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1613,
"max" : 61526,
"mean" : 2852.995997,
"stddev" : 1262.832611,
"N" : 1975073
},
"clat_ns" : {
"min" : 631,
"max" : 118362,
"mean" : 8518.367863,
"stddev" : 2664.843579,
"N" : 1975072,
"percentile" : {
"1.000000" : 4832,
"5.000000" : 4960,
"10.000000" : 5088,
"20.000000" : 6880,
"30.000000" : 7456,
"40.000000" : 7584,
"50.000000" : 7712,
"60.000000" : 9280,
"70.000000" : 9408,
"80.000000" : 9536,
"90.000000" : 10944,
"95.000000" : 12224,
"99.000000" : 18304,
"99.500000" : 20864,
"99.900000" : 24960,
"99.950000" : 30592,
"99.990000" : 62208
}
},
"lat_ns" : {
"min" : 6021,
"max" : 120075,
"mean" : 11371.363823,
"stddev" : 3297.605000,
"N" : 1975072
},
"bw_min" : 140440,
"bw_max" : 142061,
"bw_agg" : 100.000000,
"bw_mean" : 141095.714286,
"bw_dev" : 440.312377,
"bw_samples" : 14,
"iops_min" : 280880,
"iops_max" : 284122,
"iops_mean" : 282191.428571,
"iops_stddev" : 880.624753,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 43.471429,
"sys_cpu" : 56.500000,
"ctx" : 20,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.861437,
"2" : 55.847635,
"4" : 43.290928,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.052150,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.018075,
"4" : 0.862348,
"10" : 82.898092,
"20" : 15.549256,
"50" : 0.580789,
"100" : 0.035391,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142860: Wed Oct 26 04:45:03 2022
read: IOPS=282k, BW=138MiB/s (144MB/s)(964MiB/7001msec)
slat (nsec): min=1613, max=61526, avg=2853.00, stdev=1262.83
clat (nsec): min=631, max=118362, avg=8518.37, stdev=2664.84
lat (usec): min=6, max=120, avg=11.37, stdev= 3.30
clat percentiles (nsec):
| 1.00th=[ 4832], 5.00th=[ 4960], 10.00th=[ 5088], 20.00th=[ 6880],
| 30.00th=[ 7456], 40.00th=[ 7584], 50.00th=[ 7712], 60.00th=[ 9280],
| 70.00th=[ 9408], 80.00th=[ 9536], 90.00th=[10944], 95.00th=[12224],
| 99.00th=[18304], 99.50th=[20864], 99.90th=[24960], 99.95th=[30592],
| 99.99th=[62208]
bw ( KiB/s): min=140440, max=142061, per=100.00%, avg=141095.71, stdev=440.31, samples=14
iops : min=280880, max=284122, avg=282191.43, stdev=880.62, samples=14
lat (nsec) : 750=0.05%, 1000=0.01%
lat (usec) : 2=0.02%, 4=0.86%, 10=82.90%, 20=15.55%, 50=0.58%
lat (usec) : 100=0.04%, 250=0.01%
cpu : usr=43.47%, sys=56.50%, ctx=20, majf=0, minf=0
IO depths : 1=0.9%, 2=55.8%, 4=43.3%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1975072,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=138MiB/s (144MB/s), 138MiB/s-138MiB/s (144MB/s-144MB/s), io=964MiB (1011MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_0.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752317,
"timestamp_ms" : 1666752317061,
"time" : "Wed Oct 26 04:45:17 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 995397120,
"io_kbytes" : 972067,
"bw_bytes" : 142179277,
"bw" : 138846,
"iops" : 277693.329524,
"runtime" : 7001,
"total_ios" : 1944131,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1472,
"max" : 23805,
"mean" : 2275.900375,
"stddev" : 900.413544,
"N" : 1944132
},
"clat_ns" : {
"min" : 4949,
"max" : 133701,
"mean" : 11331.964851,
"stddev" : 2845.799578,
"N" : 1944134,
"percentile" : {
"1.000000" : 9408,
"5.000000" : 9536,
"10.000000" : 9536,
"20.000000" : 9664,
"30.000000" : 9664,
"40.000000" : 9792,
"50.000000" : 11072,
"60.000000" : 12096,
"70.000000" : 12224,
"80.000000" : 12352,
"90.000000" : 12352,
"95.000000" : 12608,
"99.000000" : 23680,
"99.500000" : 28288,
"99.900000" : 36096,
"99.950000" : 40192,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 7484,
"max" : 136586,
"mean" : 13607.865899,
"stddev" : 2632.972492,
"N" : 1944134
},
"bw_min" : 138554,
"bw_max" : 139440,
"bw_agg" : 100.000000,
"bw_mean" : 138885.857143,
"bw_dev" : 275.370200,
"bw_samples" : 14,
"iops_min" : 277108,
"iops_max" : 278880,
"iops_mean" : 277771.785714,
"iops_stddev" : 550.705586,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 36.685714,
"sys_cpu" : 61.342857,
"ctx" : 11813,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 45.621154,
"20" : 52.590952,
"50" : 1.747002,
"100" : 0.039349,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=142899: Wed Oct 26 04:45:17 2022
read: IOPS=278k, BW=136MiB/s (142MB/s)(949MiB/7001msec)
slat (nsec): min=1472, max=23805, avg=2275.90, stdev=900.41
clat (usec): min=4, max=133, avg=11.33, stdev= 2.85
lat (usec): min=7, max=136, avg=13.61, stdev= 2.63
clat percentiles (nsec):
| 1.00th=[ 9408], 5.00th=[ 9536], 10.00th=[ 9536], 20.00th=[ 9664],
| 30.00th=[ 9664], 40.00th=[ 9792], 50.00th=[11072], 60.00th=[12096],
| 70.00th=[12224], 80.00th=[12352], 90.00th=[12352], 95.00th=[12608],
| 99.00th=[23680], 99.50th=[28288], 99.90th=[36096], 99.95th=[40192],
| 99.99th=[72192]
bw ( KiB/s): min=138554, max=139440, per=100.00%, avg=138885.86, stdev=275.37, samples=14
iops : min=277108, max=278880, avg=277771.79, stdev=550.71, samples=14
lat (usec) : 10=45.62%, 20=52.59%, 50=1.75%, 100=0.04%, 250=0.01%
cpu : usr=36.69%, sys=61.34%, ctx=11813, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1944131,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=136MiB/s (142MB/s), 136MiB/s-136MiB/s (142MB/s-142MB/s), io=949MiB (995MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752330,
"timestamp_ms" : 1666752330841,
"time" : "Wed Oct 26 04:45:30 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 998190080,
"io_kbytes" : 974795,
"bw_bytes" : 142578214,
"bw" : 139236,
"iops" : 278472.503928,
"runtime" : 7001,
"total_ios" : 1949586,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1462,
"max" : 34996,
"mean" : 2286.772988,
"stddev" : 911.175884,
"N" : 1949587
},
"clat_ns" : {
"min" : 501,
"max" : 134612,
"mean" : 11299.563884,
"stddev" : 2830.245084,
"N" : 1949589,
"percentile" : {
"1.000000" : 9408,
"5.000000" : 9536,
"10.000000" : 9536,
"20.000000" : 9536,
"30.000000" : 9664,
"40.000000" : 9664,
"50.000000" : 10944,
"60.000000" : 12096,
"70.000000" : 12224,
"80.000000" : 12224,
"90.000000" : 12352,
"95.000000" : 12608,
"99.000000" : 23680,
"99.500000" : 28288,
"99.900000" : 36096,
"99.950000" : 41216,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 8206,
"max" : 138840,
"mean" : 13586.337374,
"stddev" : 2615.938007,
"N" : 1949589
},
"bw_min" : 138799,
"bw_max" : 139789,
"bw_agg" : 100.000000,
"bw_mean" : 139298.538462,
"bw_dev" : 326.805145,
"bw_samples" : 13,
"iops_min" : 277600,
"iops_max" : 279578,
"iops_mean" : 278597.230769,
"iops_stddev" : 653.355717,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 38.457143,
"sys_cpu" : 59.557143,
"ctx" : 11881,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 45.654565,
"20" : 52.723501,
"50" : 1.577668,
"100" : 0.043599,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=142939: Wed Oct 26 04:45:30 2022
read: IOPS=278k, BW=136MiB/s (143MB/s)(952MiB/7001msec)
slat (nsec): min=1462, max=34996, avg=2286.77, stdev=911.18
clat (nsec): min=501, max=134612, avg=11299.56, stdev=2830.25
lat (usec): min=8, max=138, avg=13.59, stdev= 2.62
clat percentiles (nsec):
| 1.00th=[ 9408], 5.00th=[ 9536], 10.00th=[ 9536], 20.00th=[ 9536],
| 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[10944], 60.00th=[12096],
| 70.00th=[12224], 80.00th=[12224], 90.00th=[12352], 95.00th=[12608],
| 99.00th=[23680], 99.50th=[28288], 99.90th=[36096], 99.95th=[41216],
| 99.99th=[72192]
bw ( KiB/s): min=138799, max=139789, per=100.00%, avg=139298.54, stdev=326.81, samples=13
iops : min=277600, max=279578, avg=278597.23, stdev=653.36, samples=13
lat (nsec) : 750=0.01%
lat (usec) : 10=45.65%, 20=52.72%, 50=1.58%, 100=0.04%, 250=0.01%
cpu : usr=38.46%, sys=59.56%, ctx=11881, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1949586,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=952MiB (998MB), run=7001-7001msec
$ rm /tmp/fio-output_512_4_libaio-xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752344,
"timestamp_ms" : 1666752344512,
"time" : "Wed Oct 26 04:45:44 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "4",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1003818496,
"io_kbytes" : 980291,
"bw_bytes" : 143382159,
"bw" : 140021,
"iops" : 280042.708185,
"runtime" : 7001,
"total_ios" : 1960579,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1462,
"max" : 27762,
"mean" : 2373.694871,
"stddev" : 1047.956777,
"N" : 1960580
},
"clat_ns" : {
"min" : 4919,
"max" : 134282,
"mean" : 11239.872907,
"stddev" : 2798.284062,
"N" : 1960582,
"percentile" : {
"1.000000" : 9408,
"5.000000" : 9408,
"10.000000" : 9408,
"20.000000" : 9536,
"30.000000" : 9536,
"40.000000" : 9664,
"50.000000" : 10944,
"60.000000" : 11968,
"70.000000" : 12096,
"80.000000" : 12224,
"90.000000" : 12352,
"95.000000" : 12608,
"99.000000" : 23680,
"99.500000" : 28288,
"99.900000" : 36096,
"99.950000" : 40192,
"99.990000" : 70144
}
},
"lat_ns" : {
"min" : 7174,
"max" : 138400,
"mean" : 13613.568208,
"stddev" : 2576.989435,
"N" : 1960582
},
"bw_min" : 139649,
"bw_max" : 140398,
"bw_agg" : 100.000000,
"bw_mean" : 140066.571429,
"bw_dev" : 220.341593,
"bw_samples" : 14,
"iops_min" : 279298,
"iops_max" : 280796,
"iops_mean" : 280133.214286,
"iops_stddev" : 440.635935,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 38.142857,
"sys_cpu" : 59.885714,
"ctx" : 11851,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 45.917150,
"20" : 52.538204,
"50" : 1.504913,
"100" : 0.039019,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 4,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=142978: Wed Oct 26 04:45:44 2022
read: IOPS=280k, BW=137MiB/s (143MB/s)(957MiB/7001msec)
slat (nsec): min=1462, max=27762, avg=2373.69, stdev=1047.96
clat (usec): min=4, max=134, avg=11.24, stdev= 2.80
lat (usec): min=7, max=138, avg=13.61, stdev= 2.58
clat percentiles (nsec):
| 1.00th=[ 9408], 5.00th=[ 9408], 10.00th=[ 9408], 20.00th=[ 9536],
| 30.00th=[ 9536], 40.00th=[ 9664], 50.00th=[10944], 60.00th=[11968],
| 70.00th=[12096], 80.00th=[12224], 90.00th=[12352], 95.00th=[12608],
| 99.00th=[23680], 99.50th=[28288], 99.90th=[36096], 99.95th=[40192],
| 99.99th=[70144]
bw ( KiB/s): min=139649, max=140398, per=100.00%, avg=140066.57, stdev=220.34, samples=14
iops : min=279298, max=280796, avg=280133.21, stdev=440.64, samples=14
lat (usec) : 10=45.92%, 20=52.54%, 50=1.50%, 100=0.04%, 250=0.01%
cpu : usr=38.14%, sys=59.89%, ctx=11851, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1960579,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=957MiB (1004MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751067,
"timestamp_ms" : 1666751067207,
"time" : "Wed Oct 26 04:24:27 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 282276352,
"io_kbytes" : 275660,
"bw_bytes" : 40319433,
"bw" : 39374,
"iops" : 78748.893015,
"runtime" : 7001,
"total_ios" : 551321,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2504,
"max" : 49713,
"mean" : 2754.340312,
"stddev" : 211.568405,
"N" : 551321
},
"clat_ns" : {
"min" : 200,
"max" : 99306,
"mean" : 9273.575117,
"stddev" : 1277.471589,
"N" : 551321,
"percentile" : {
"1.000000" : 8896,
"5.000000" : 8896,
"10.000000" : 9024,
"20.000000" : 9024,
"30.000000" : 9024,
"40.000000" : 9024,
"50.000000" : 9152,
"60.000000" : 9152,
"70.000000" : 9152,
"80.000000" : 9152,
"90.000000" : 9280,
"95.000000" : 9536,
"99.000000" : 15168,
"99.500000" : 15808,
"99.900000" : 19840,
"99.950000" : 26240,
"99.990000" : 56064
}
},
"lat_ns" : {
"min" : 9197,
"max" : 102102,
"mean" : 12027.915429,
"stddev" : 1296.149259,
"N" : 551321
},
"bw_min" : 39234,
"bw_max" : 39603,
"bw_agg" : 100.000000,
"bw_mean" : 39390.142857,
"bw_dev" : 106.081142,
"bw_samples" : 14,
"iops_min" : 78468,
"iops_max" : 79206,
"iops_mean" : 78780.357143,
"iops_stddev" : 212.094176,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 12.414286,
"sys_cpu" : 48.628571,
"ctx" : 551215,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.031312,
"20" : 2.883982,
"50" : 0.057317,
"100" : 0.019408,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 793290,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5015,
"write_ticks" : 0,
"in_queue" : 5015,
"util" : 99.049128
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=139058: Wed Oct 26 04:24:27 2022
read: IOPS=78.7k, BW=38.5MiB/s (40.3MB/s)(269MiB/7001msec)
slat (nsec): min=2504, max=49713, avg=2754.34, stdev=211.57
clat (nsec): min=200, max=99306, avg=9273.58, stdev=1277.47
lat (usec): min=9, max=102, avg=12.03, stdev= 1.30
clat percentiles (nsec):
| 1.00th=[ 8896], 5.00th=[ 8896], 10.00th=[ 9024], 20.00th=[ 9024],
| 30.00th=[ 9024], 40.00th=[ 9024], 50.00th=[ 9152], 60.00th=[ 9152],
| 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9536],
| 99.00th=[15168], 99.50th=[15808], 99.90th=[19840], 99.95th=[26240],
| 99.99th=[56064]
bw ( KiB/s): min=39234, max=39603, per=100.00%, avg=39390.14, stdev=106.08, samples=14
iops : min=78468, max=79206, avg=78780.36, stdev=212.09, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 10=97.03%, 20=2.88%, 50=0.06%, 100=0.02%
cpu : usr=12.41%, sys=48.63%, ctx=551215, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=551321,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=38.5MiB/s (40.3MB/s), 38.5MiB/s-38.5MiB/s (40.3MB/s-40.3MB/s), io=269MiB (282MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=793290/0, merge=0/0, ticks=5015/0, in_queue=5015, util=99.05%
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752358,
"timestamp_ms" : 1666752358142,
"time" : "Wed Oct 26 04:45:58 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1195644416,
"io_kbytes" : 1167621,
"bw_bytes" : 170781947,
"bw" : 166779,
"iops" : 333557.491787,
"runtime" : 7001,
"total_ios" : 2335236,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1152,
"max" : 19917,
"mean" : 2244.668577,
"stddev" : 1068.788020,
"N" : 2335236
},
"clat_ns" : {
"min" : 80,
"max" : 145854,
"mean" : 21249.490787,
"stddev" : 3863.814320,
"N" : 2335243,
"percentile" : {
"1.000000" : 13760,
"5.000000" : 18816,
"10.000000" : 19072,
"20.000000" : 19328,
"30.000000" : 20352,
"40.000000" : 20608,
"50.000000" : 20608,
"60.000000" : 20864,
"70.000000" : 21376,
"80.000000" : 21888,
"90.000000" : 22656,
"95.000000" : 24960,
"99.000000" : 41728,
"99.500000" : 44288,
"99.900000" : 56064,
"99.950000" : 67072,
"99.990000" : 82432
}
},
"lat_ns" : {
"min" : 6612,
"max" : 147096,
"mean" : 23494.157402,
"stddev" : 3821.091989,
"N" : 2335243
},
"bw_min" : 166104,
"bw_max" : 167290,
"bw_agg" : 100.000000,
"bw_mean" : 166794.153846,
"bw_dev" : 286.537329,
"bw_samples" : 13,
"iops_min" : 332210,
"iops_max" : 334580,
"iops_mean" : 333588.461538,
"iops_stddev" : 572.567989,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.142857,
"sys_cpu" : 83.414286,
"ctx" : 12659,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999957,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.016915,
"20" : 22.109457,
"50" : 77.740494,
"100" : 0.130993,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=143018: Wed Oct 26 04:45:58 2022
read: IOPS=334k, BW=163MiB/s (171MB/s)(1140MiB/7001msec)
slat (nsec): min=1152, max=19917, avg=2244.67, stdev=1068.79
clat (nsec): min=80, max=145854, avg=21249.49, stdev=3863.81
lat (usec): min=6, max=147, avg=23.49, stdev= 3.82
clat percentiles (nsec):
| 1.00th=[13760], 5.00th=[18816], 10.00th=[19072], 20.00th=[19328],
| 30.00th=[20352], 40.00th=[20608], 50.00th=[20608], 60.00th=[20864],
| 70.00th=[21376], 80.00th=[21888], 90.00th=[22656], 95.00th=[24960],
| 99.00th=[41728], 99.50th=[44288], 99.90th=[56064], 99.95th=[67072],
| 99.99th=[82432]
bw ( KiB/s): min=166104, max=167290, per=100.00%, avg=166794.15, stdev=286.54, samples=13
iops : min=332210, max=334580, avg=333588.46, stdev=572.57, samples=13
lat (nsec) : 100=0.01%
lat (usec) : 4=0.01%, 10=0.02%, 20=22.11%, 50=77.74%, 100=0.13%
lat (usec) : 250=0.01%
cpu : usr=14.14%, sys=83.41%, ctx=12659, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2335236,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=1140MiB (1196MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752371,
"timestamp_ms" : 1666752371752,
"time" : "Wed Oct 26 04:46:11 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1196563968,
"io_kbytes" : 1168519,
"bw_bytes" : 170913293,
"bw" : 166907,
"iops" : 333813.883731,
"runtime" : 7001,
"total_ios" : 2337031,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1152,
"max" : 24005,
"mean" : 2220.236975,
"stddev" : 1090.837490,
"N" : 2337032
},
"clat_ns" : {
"min" : 3196,
"max" : 144360,
"mean" : 21244.316029,
"stddev" : 3839.831790,
"N" : 2337038,
"percentile" : {
"1.000000" : 13888,
"5.000000" : 18816,
"10.000000" : 19072,
"20.000000" : 19328,
"30.000000" : 20352,
"40.000000" : 20608,
"50.000000" : 20608,
"60.000000" : 20864,
"70.000000" : 21632,
"80.000000" : 21888,
"90.000000" : 22912,
"95.000000" : 24960,
"99.000000" : 41728,
"99.500000" : 44288,
"99.900000" : 53504,
"99.950000" : 64768,
"99.990000" : 81408
}
},
"lat_ns" : {
"min" : 7093,
"max" : 145563,
"mean" : 23464.552185,
"stddev" : 3807.683431,
"N" : 2337038
},
"bw_min" : 166713,
"bw_max" : 167275,
"bw_agg" : 100.000000,
"bw_mean" : 166953.357143,
"bw_dev" : 172.887517,
"bw_samples" : 14,
"iops_min" : 333426,
"iops_max" : 334550,
"iops_mean" : 333906.928571,
"iops_stddev" : 345.829763,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.800000,
"sys_cpu" : 83.771429,
"ctx" : 12624,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999957,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.015575,
"20" : 23.539739,
"50" : 76.322522,
"100" : 0.120965,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=143058: Wed Oct 26 04:46:11 2022
read: IOPS=334k, BW=163MiB/s (171MB/s)(1141MiB/7001msec)
slat (nsec): min=1152, max=24005, avg=2220.24, stdev=1090.84
clat (usec): min=3, max=144, avg=21.24, stdev= 3.84
lat (usec): min=7, max=145, avg=23.46, stdev= 3.81
clat percentiles (nsec):
| 1.00th=[13888], 5.00th=[18816], 10.00th=[19072], 20.00th=[19328],
| 30.00th=[20352], 40.00th=[20608], 50.00th=[20608], 60.00th=[20864],
| 70.00th=[21632], 80.00th=[21888], 90.00th=[22912], 95.00th=[24960],
| 99.00th=[41728], 99.50th=[44288], 99.90th=[53504], 99.95th=[64768],
| 99.99th=[81408]
bw ( KiB/s): min=166713, max=167275, per=100.00%, avg=166953.36, stdev=172.89, samples=14
iops : min=333426, max=334550, avg=333906.93, stdev=345.83, samples=14
lat (usec) : 4=0.01%, 10=0.02%, 20=23.54%, 50=76.32%, 100=0.12%
lat (usec) : 250=0.01%
cpu : usr=13.80%, sys=83.77%, ctx=12624, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2337031,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=1141MiB (1197MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752385,
"timestamp_ms" : 1666752385359,
"time" : "Wed Oct 26 04:46:25 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring_cmd",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1210347520,
"io_kbytes" : 1181980,
"bw_bytes" : 172882091,
"bw" : 168830,
"iops" : 337659.334381,
"runtime" : 7001,
"total_ios" : 2363953,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1122,
"max" : 22653,
"mean" : 2034.197053,
"stddev" : 1011.133446,
"N" : 2363953
},
"clat_ns" : {
"min" : 90,
"max" : 142347,
"mean" : 21168.681512,
"stddev" : 3823.819840,
"N" : 2363960,
"percentile" : {
"1.000000" : 14400,
"5.000000" : 18560,
"10.000000" : 18816,
"20.000000" : 19072,
"30.000000" : 20352,
"40.000000" : 20352,
"50.000000" : 20608,
"60.000000" : 21120,
"70.000000" : 21632,
"80.000000" : 21888,
"90.000000" : 22912,
"95.000000" : 24704,
"99.000000" : 41216,
"99.500000" : 43776,
"99.900000" : 55040,
"99.950000" : 66048,
"99.990000" : 80384
}
},
"lat_ns" : {
"min" : 5551,
"max" : 143539,
"mean" : 23202.878542,
"stddev" : 3787.397119,
"N" : 2363960
},
"bw_min" : 168266,
"bw_max" : 169275,
"bw_agg" : 100.000000,
"bw_mean" : 168882.153846,
"bw_dev" : 336.167579,
"bw_samples" : 13,
"iops_min" : 336532,
"iops_max" : 338550,
"iops_mean" : 337764.307692,
"iops_stddev" : 672.444221,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.700000,
"sys_cpu" : 83.814286,
"ctx" : 12774,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999958,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 0.013790,
"20" : 23.539004,
"50" : 76.321145,
"100" : 0.124791,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=143096: Wed Oct 26 04:46:25 2022
read: IOPS=338k, BW=165MiB/s (173MB/s)(1154MiB/7001msec)
slat (nsec): min=1122, max=22653, avg=2034.20, stdev=1011.13
clat (nsec): min=90, max=142347, avg=21168.68, stdev=3823.82
lat (usec): min=5, max=143, avg=23.20, stdev= 3.79
clat percentiles (nsec):
| 1.00th=[14400], 5.00th=[18560], 10.00th=[18816], 20.00th=[19072],
| 30.00th=[20352], 40.00th=[20352], 50.00th=[20608], 60.00th=[21120],
| 70.00th=[21632], 80.00th=[21888], 90.00th=[22912], 95.00th=[24704],
| 99.00th=[41216], 99.50th=[43776], 99.90th=[55040], 99.95th=[66048],
| 99.99th=[80384]
bw ( KiB/s): min=168266, max=169275, per=100.00%, avg=168882.15, stdev=336.17, samples=13
iops : min=336532, max=338550, avg=337764.31, stdev=672.44, samples=13
lat (nsec) : 100=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=23.54%, 50=76.32%, 100=0.12%
lat (usec) : 250=0.01%
cpu : usr=13.70%, sys=83.81%, ctx=12774, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2363953,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=1154MiB (1210MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752398,
"timestamp_ms" : 1666752398992,
"time" : "Wed Oct 26 04:46:38 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1159592448,
"io_kbytes" : 1132414,
"bw_bytes" : 165632402,
"bw" : 161750,
"iops" : 323499.642908,
"runtime" : 7001,
"total_ios" : 2264821,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1212,
"max" : 32892,
"mean" : 2529.842174,
"stddev" : 737.661078,
"N" : 2264822
},
"clat_ns" : {
"min" : 120,
"max" : 143188,
"mean" : 21923.724910,
"stddev" : 3812.563652,
"N" : 2264828,
"percentile" : {
"1.000000" : 14272,
"5.000000" : 19840,
"10.000000" : 20352,
"20.000000" : 20864,
"30.000000" : 21120,
"40.000000" : 21120,
"50.000000" : 21376,
"60.000000" : 21376,
"70.000000" : 21632,
"80.000000" : 21888,
"90.000000" : 22912,
"95.000000" : 25984,
"99.000000" : 42240,
"99.500000" : 44800,
"99.900000" : 55040,
"99.950000" : 67072,
"99.990000" : 81408
}
},
"lat_ns" : {
"min" : 6672,
"max" : 144431,
"mean" : 24453.567314,
"stddev" : 3663.939270,
"N" : 2264828
},
"bw_min" : 161462,
"bw_max" : 162202,
"bw_agg" : 100.000000,
"bw_mean" : 161798.928571,
"bw_dev" : 238.197870,
"bw_samples" : 14,
"iops_min" : 322924,
"iops_max" : 324404,
"iops_mean" : 323597.928571,
"iops_stddev" : 476.383889,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.985714,
"sys_cpu" : 81.714286,
"ctx" : 12576,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999978,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.018721,
"20" : 6.321383,
"50" : 93.517589,
"100" : 0.139349,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=143134: Wed Oct 26 04:46:38 2022
read: IOPS=323k, BW=158MiB/s (166MB/s)(1106MiB/7001msec)
slat (nsec): min=1212, max=32892, avg=2529.84, stdev=737.66
clat (nsec): min=120, max=143188, avg=21923.72, stdev=3812.56
lat (usec): min=6, max=144, avg=24.45, stdev= 3.66
clat percentiles (nsec):
| 1.00th=[14272], 5.00th=[19840], 10.00th=[20352], 20.00th=[20864],
| 30.00th=[21120], 40.00th=[21120], 50.00th=[21376], 60.00th=[21376],
| 70.00th=[21632], 80.00th=[21888], 90.00th=[22912], 95.00th=[25984],
| 99.00th=[42240], 99.50th=[44800], 99.90th=[55040], 99.95th=[67072],
| 99.99th=[81408]
bw ( KiB/s): min=161462, max=162202, per=100.00%, avg=161798.93, stdev=238.20, samples=14
iops : min=322924, max=324404, avg=323597.93, stdev=476.38, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=6.32%, 50=93.52%
lat (usec) : 100=0.14%, 250=0.01%
cpu : usr=15.99%, sys=81.71%, ctx=12576, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2264821,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=1106MiB (1160MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-reference_1.txt': No such file or directory
$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752412,
"timestamp_ms" : 1666752412607,
"time" : "Wed Oct 26 04:46:52 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1175407104,
"io_kbytes" : 1147858,
"bw_bytes" : 167891316,
"bw" : 163956,
"iops" : 327911.584059,
"runtime" : 7001,
"total_ios" : 2295709,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1202,
"max" : 46768,
"mean" : 2507.877895,
"stddev" : 718.916471,
"N" : 2295710
},
"clat_ns" : {
"min" : 120,
"max" : 143369,
"mean" : 21623.022268,
"stddev" : 3822.739674,
"N" : 2295716,
"percentile" : {
"1.000000" : 14144,
"5.000000" : 19840,
"10.000000" : 20608,
"20.000000" : 20608,
"30.000000" : 20864,
"40.000000" : 20864,
"50.000000" : 21120,
"60.000000" : 21120,
"70.000000" : 21120,
"80.000000" : 21376,
"90.000000" : 22656,
"95.000000" : 25728,
"99.000000" : 42240,
"99.500000" : 44288,
"99.900000" : 56064,
"99.950000" : 68096,
"99.990000" : 83456
}
},
"lat_ns" : {
"min" : 6823,
"max" : 144632,
"mean" : 24130.900482,
"stddev" : 3672.577138,
"N" : 2295716
},
"bw_min" : 163612,
"bw_max" : 164316,
"bw_agg" : 100.000000,
"bw_mean" : 164015.000000,
"bw_dev" : 228.698856,
"bw_samples" : 13,
"iops_min" : 327224,
"iops_max" : 328633,
"iops_mean" : 328030.230769,
"iops_stddev" : 457.340893,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.928571,
"sys_cpu" : 81.728571,
"ctx" : 12743,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999978,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.016378,
"20" : 6.622529,
"50" : 93.218435,
"100" : 0.138563,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=143172: Wed Oct 26 04:46:52 2022
read: IOPS=328k, BW=160MiB/s (168MB/s)(1121MiB/7001msec)
slat (nsec): min=1202, max=46768, avg=2507.88, stdev=718.92
clat (nsec): min=120, max=143369, avg=21623.02, stdev=3822.74
lat (usec): min=6, max=144, avg=24.13, stdev= 3.67
clat percentiles (nsec):
| 1.00th=[14144], 5.00th=[19840], 10.00th=[20608], 20.00th=[20608],
| 30.00th=[20864], 40.00th=[20864], 50.00th=[21120], 60.00th=[21120],
| 70.00th=[21120], 80.00th=[21376], 90.00th=[22656], 95.00th=[25728],
| 99.00th=[42240], 99.50th=[44288], 99.90th=[56064], 99.95th=[68096],
| 99.99th=[83456]
bw ( KiB/s): min=163612, max=164316, per=100.00%, avg=164015.00, stdev=228.70, samples=13
iops : min=327224, max=328633, avg=328030.23, stdev=457.34, samples=13
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=6.62%, 50=93.22%
lat (usec) : 100=0.14%, 250=0.01%
cpu : usr=15.93%, sys=81.73%, ctx=12743, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2295709,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=160MiB/s (168MB/s), 160MiB/s-160MiB/s (168MB/s-168MB/s), io=1121MiB (1175MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752426,
"timestamp_ms" : 1666752426234,
"time" : "Wed Oct 26 04:47:06 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring_cmd",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/ng0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1161004544,
"io_kbytes" : 1133793,
"bw_bytes" : 165834101,
"bw" : 161947,
"iops" : 323893.586630,
"runtime" : 7001,
"total_ios" : 2267579,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1202,
"max" : 38503,
"mean" : 2508.514283,
"stddev" : 748.390671,
"N" : 2267580
},
"clat_ns" : {
"min" : 121,
"max" : 141366,
"mean" : 21924.472038,
"stddev" : 3826.780599,
"N" : 2267586,
"percentile" : {
"1.000000" : 14272,
"5.000000" : 19840,
"10.000000" : 20352,
"20.000000" : 20864,
"30.000000" : 21120,
"40.000000" : 21120,
"50.000000" : 21376,
"60.000000" : 21376,
"70.000000" : 21632,
"80.000000" : 22144,
"90.000000" : 22912,
"95.000000" : 25728,
"99.000000" : 42752,
"99.500000" : 44800,
"99.900000" : 56576,
"99.950000" : 67072,
"99.990000" : 81408
}
},
"lat_ns" : {
"min" : 6502,
"max" : 142618,
"mean" : 24432.986819,
"stddev" : 3677.160613,
"N" : 2267586
},
"bw_min" : 161484,
"bw_max" : 162868,
"bw_agg" : 100.000000,
"bw_mean" : 161985.571429,
"bw_dev" : 433.250099,
"bw_samples" : 14,
"iops_min" : 322966,
"iops_max" : 325736,
"iops_mean" : 323971.214286,
"iops_stddev" : 866.505997,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.057143,
"sys_cpu" : 80.628571,
"ctx" : 12620,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999978,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.018875,
"20" : 6.020430,
"50" : 93.815871,
"100" : 0.142884,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=143210: Wed Oct 26 04:47:06 2022
read: IOPS=324k, BW=158MiB/s (166MB/s)(1107MiB/7001msec)
slat (nsec): min=1202, max=38503, avg=2508.51, stdev=748.39
clat (nsec): min=121, max=141366, avg=21924.47, stdev=3826.78
lat (usec): min=6, max=142, avg=24.43, stdev= 3.68
clat percentiles (nsec):
| 1.00th=[14272], 5.00th=[19840], 10.00th=[20352], 20.00th=[20864],
| 30.00th=[21120], 40.00th=[21120], 50.00th=[21376], 60.00th=[21376],
| 70.00th=[21632], 80.00th=[22144], 90.00th=[22912], 95.00th=[25728],
| 99.00th=[42752], 99.50th=[44800], 99.90th=[56576], 99.95th=[67072],
| 99.99th=[81408]
bw ( KiB/s): min=161484, max=162868, per=100.00%, avg=161985.57, stdev=433.25, samples=14
iops : min=322966, max=325736, avg=323971.21, stdev=866.51, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=6.02%, 50=93.82%
lat (usec) : 100=0.14%, 250=0.01%
cpu : usr=17.06%, sys=80.63%, ctx=12620, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2267579,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=1107MiB (1161MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752441,
"timestamp_ms" : 1666752441259,
"time" : "Wed Oct 26 04:47:21 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 2276859392,
"io_kbytes" : 2223495,
"bw_bytes" : 325219167,
"bw" : 317596,
"iops" : 635192.543922,
"runtime" : 7001,
"total_ios" : 4446983,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 190,
"max" : 30808,
"mean" : 266.992913,
"stddev" : 64.891956,
"N" : 4446984
},
"clat_ns" : {
"min" : 441,
"max" : 133430,
"mean" : 11928.421873,
"stddev" : 4431.847865,
"N" : 4446983,
"percentile" : {
"1.000000" : 7008,
"5.000000" : 7904,
"10.000000" : 8512,
"20.000000" : 9536,
"30.000000" : 10048,
"40.000000" : 10560,
"50.000000" : 10944,
"60.000000" : 11584,
"70.000000" : 12224,
"80.000000" : 13120,
"90.000000" : 14784,
"95.000000" : 19840,
"99.000000" : 31104,
"99.500000" : 34560,
"99.900000" : 47872,
"99.950000" : 65280,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 5991,
"max" : 133711,
"mean" : 12195.414785,
"stddev" : 4429.865740,
"N" : 4446983
},
"bw_min" : 317209,
"bw_max" : 318994,
"bw_agg" : 100.000000,
"bw_mean" : 317838.615385,
"bw_dev" : 507.818461,
"bw_samples" : 13,
"iops_min" : 634418,
"iops_max" : 637988,
"iops_mean" : 635677.538462,
"iops_stddev" : 1015.420079,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 24.005084,
"8" : 75.986191,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.992103,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 28.089853,
"20" : 66.951931,
"50" : 4.864332,
"100" : 0.091613,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143259: Wed Oct 26 04:47:21 2022
read: IOPS=635k, BW=310MiB/s (325MB/s)(2171MiB/7001msec)
slat (nsec): min=190, max=30808, avg=266.99, stdev=64.89
clat (nsec): min=441, max=133430, avg=11928.42, stdev=4431.85
lat (usec): min=5, max=133, avg=12.20, stdev= 4.43
clat percentiles (nsec):
| 1.00th=[ 7008], 5.00th=[ 7904], 10.00th=[ 8512], 20.00th=[ 9536],
| 30.00th=[10048], 40.00th=[10560], 50.00th=[10944], 60.00th=[11584],
| 70.00th=[12224], 80.00th=[13120], 90.00th=[14784], 95.00th=[19840],
| 99.00th=[31104], 99.50th=[34560], 99.90th=[47872], 99.95th=[65280],
| 99.99th=[79360]
bw ( KiB/s): min=317209, max=318994, per=100.00%, avg=317838.62, stdev=507.82, samples=13
iops : min=634418, max=637988, avg=635677.54, stdev=1015.42, samples=13
lat (nsec) : 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=28.09%, 20=66.95%, 50=4.86%
lat (usec) : 100=0.09%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=22, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=24.0%, 8=76.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4446983,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=310MiB/s (325MB/s), 310MiB/s-310MiB/s (325MB/s-325MB/s), io=2171MiB (2277MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_1.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752456,
"timestamp_ms" : 1666752456347,
"time" : "Wed Oct 26 04:47:36 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1845574656,
"io_kbytes" : 1802319,
"bw_bytes" : 263615862,
"bw" : 257437,
"iops" : 514873.589487,
"runtime" : 7001,
"total_ios" : 3604630,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 170,
"max" : 9297,
"mean" : 249.728569,
"stddev" : 28.581610,
"N" : 3604631
},
"clat_ns" : {
"min" : 4809,
"max" : 130585,
"mean" : 14363.971523,
"stddev" : 5215.015737,
"N" : 3604630,
"percentile" : {
"1.000000" : 7456,
"5.000000" : 8384,
"10.000000" : 8896,
"20.000000" : 9792,
"30.000000" : 10560,
"40.000000" : 11712,
"50.000000" : 14144,
"60.000000" : 15424,
"70.000000" : 16512,
"80.000000" : 18560,
"90.000000" : 20096,
"95.000000" : 22144,
"99.000000" : 31616,
"99.500000" : 35584,
"99.900000" : 48384,
"99.950000" : 62720,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 5080,
"max" : 130805,
"mean" : 14613.700084,
"stddev" : 5209.006901,
"N" : 3604630
},
"bw_min" : 256603,
"bw_max" : 259025,
"bw_agg" : 100.000000,
"bw_mean" : 257567.500000,
"bw_dev" : 671.520634,
"bw_samples" : 14,
"iops_min" : 513206,
"iops_max" : 518050,
"iops_mean" : 515135.142857,
"iops_stddev" : 1342.858940,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 26,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 9.715921,
"4" : 43.387421,
"8" : 46.894910,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 95.677259,
"8" : 4.322741,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 22.285450,
"20" : 67.773863,
"50" : 9.846531,
"100" : 0.092104,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143310: Wed Oct 26 04:47:36 2022
read: IOPS=515k, BW=251MiB/s (264MB/s)(1760MiB/7001msec)
slat (nsec): min=170, max=9297, avg=249.73, stdev=28.58
clat (usec): min=4, max=130, avg=14.36, stdev= 5.22
lat (usec): min=5, max=130, avg=14.61, stdev= 5.21
clat percentiles (nsec):
| 1.00th=[ 7456], 5.00th=[ 8384], 10.00th=[ 8896], 20.00th=[ 9792],
| 30.00th=[10560], 40.00th=[11712], 50.00th=[14144], 60.00th=[15424],
| 70.00th=[16512], 80.00th=[18560], 90.00th=[20096], 95.00th=[22144],
| 99.00th=[31616], 99.50th=[35584], 99.90th=[48384], 99.95th=[62720],
| 99.99th=[78336]
bw ( KiB/s): min=256603, max=259025, per=100.00%, avg=257567.50, stdev=671.52, samples=14
iops : min=513206, max=518050, avg=515135.14, stdev=1342.86, samples=14
lat (usec) : 10=22.29%, 20=67.77%, 50=9.85%, 100=0.09%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=26, majf=0, minf=0
IO depths : 1=0.1%, 2=9.7%, 4=43.4%, 8=46.9%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.7%, 8=4.3%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3604630,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=251MiB/s (264MB/s), 251MiB/s-251MiB/s (264MB/s-264MB/s), io=1760MiB (1846MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752471,
"timestamp_ms" : 1666752471482,
"time" : "Wed Oct 26 04:47:51 2022",
"jobs" : [
{
"jobname" : "io_uring_cmd-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring_cmd-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "0",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 2261216256,
"io_kbytes" : 2208219,
"bw_bytes" : 322984753,
"bw" : 315414,
"iops" : 630828.595915,
"runtime" : 7001,
"total_ios" : 4416431,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 190,
"max" : 24937,
"mean" : 265.485439,
"stddev" : 66.086546,
"N" : 4416431
},
"clat_ns" : {
"min" : 921,
"max" : 131396,
"mean" : 12004.528837,
"stddev" : 4455.386441,
"N" : 4416438,
"percentile" : {
"1.000000" : 6816,
"5.000000" : 7904,
"10.000000" : 8512,
"20.000000" : 9536,
"30.000000" : 10176,
"40.000000" : 10688,
"50.000000" : 11072,
"60.000000" : 11712,
"70.000000" : 12480,
"80.000000" : 13248,
"90.000000" : 14784,
"95.000000" : 19840,
"99.000000" : 31104,
"99.500000" : 34560,
"99.900000" : 49920,
"99.950000" : 67072,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 5580,
"max" : 131617,
"mean" : 12270.014258,
"stddev" : 4453.681752,
"N" : 4416438
},
"bw_min" : 314606,
"bw_max" : 317057,
"bw_agg" : 100.000000,
"bw_mean" : 315565.928571,
"bw_dev" : 753.328967,
"bw_samples" : 14,
"iops_min" : 629212,
"iops_max" : 634114,
"iops_mean" : 631131.928571,
"iops_stddev" : 1506.587610,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 99.985714,
"sys_cpu" : 0.000000,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.100000,
"2" : 0.100000,
"4" : 25.115280,
"8" : 74.875663,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.991752,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 28.039406,
"20" : 66.984359,
"50" : 4.876879,
"100" : 0.097998,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143360: Wed Oct 26 04:47:51 2022
read: IOPS=631k, BW=308MiB/s (323MB/s)(2156MiB/7001msec)
slat (nsec): min=190, max=24937, avg=265.49, stdev=66.09
clat (nsec): min=921, max=131396, avg=12004.53, stdev=4455.39
lat (usec): min=5, max=131, avg=12.27, stdev= 4.45
clat percentiles (nsec):
| 1.00th=[ 6816], 5.00th=[ 7904], 10.00th=[ 8512], 20.00th=[ 9536],
| 30.00th=[10176], 40.00th=[10688], 50.00th=[11072], 60.00th=[11712],
| 70.00th=[12480], 80.00th=[13248], 90.00th=[14784], 95.00th=[19840],
| 99.00th=[31104], 99.50th=[34560], 99.90th=[49920], 99.95th=[67072],
| 99.99th=[79360]
bw ( KiB/s): min=314606, max=317057, per=100.00%, avg=315565.93, stdev=753.33, samples=14
iops : min=629212, max=634114, avg=631131.93, stdev=1506.59, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=28.04%, 20=66.98%, 50=4.88%
lat (usec) : 100=0.10%, 250=0.01%
cpu : usr=99.99%, sys=0.00%, ctx=23, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=25.1%, 8=74.9%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4416431,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=2156MiB (2261MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752485,
"timestamp_ms" : 1666752485218,
"time" : "Wed Oct 26 04:48:05 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1124852224,
"io_kbytes" : 1098488,
"bw_bytes" : 160670221,
"bw" : 156904,
"iops" : 313807.884588,
"runtime" : 7001,
"total_ios" : 2196969,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1282,
"max" : 42510,
"mean" : 2472.162358,
"stddev" : 910.605618,
"N" : 2196970
},
"clat_ns" : {
"min" : 80,
"max" : 152536,
"mean" : 22632.731950,
"stddev" : 3858.580981,
"N" : 2196976,
"percentile" : {
"1.000000" : 15552,
"5.000000" : 20352,
"10.000000" : 20352,
"20.000000" : 20864,
"30.000000" : 21632,
"40.000000" : 21888,
"50.000000" : 22144,
"60.000000" : 22400,
"70.000000" : 23168,
"80.000000" : 23424,
"90.000000" : 23936,
"95.000000" : 26496,
"99.000000" : 42752,
"99.500000" : 45312,
"99.900000" : 54016,
"99.950000" : 66048,
"99.990000" : 81408
}
},
"lat_ns" : {
"min" : 6793,
"max" : 153869,
"mean" : 25104.895035,
"stddev" : 3768.630555,
"N" : 2196976
},
"bw_min" : 156481,
"bw_max" : 157772,
"bw_agg" : 100.000000,
"bw_mean" : 156978.923077,
"bw_dev" : 443.567819,
"bw_samples" : 13,
"iops_min" : 312962,
"iops_max" : 315544,
"iops_mean" : 313957.923077,
"iops_stddev" : 887.118036,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.042857,
"sys_cpu" : 82.785714,
"ctx" : 12432,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999954,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.022486,
"20" : 3.535780,
"50" : 96.305455,
"100" : 0.129815,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 3157749,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 17441,
"write_ticks" : 0,
"in_queue" : 17441,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=143399: Wed Oct 26 04:48:05 2022
read: IOPS=314k, BW=153MiB/s (161MB/s)(1073MiB/7001msec)
slat (nsec): min=1282, max=42510, avg=2472.16, stdev=910.61
clat (nsec): min=80, max=152536, avg=22632.73, stdev=3858.58
lat (usec): min=6, max=153, avg=25.10, stdev= 3.77
clat percentiles (nsec):
| 1.00th=[15552], 5.00th=[20352], 10.00th=[20352], 20.00th=[20864],
| 30.00th=[21632], 40.00th=[21888], 50.00th=[22144], 60.00th=[22400],
| 70.00th=[23168], 80.00th=[23424], 90.00th=[23936], 95.00th=[26496],
| 99.00th=[42752], 99.50th=[45312], 99.90th=[54016], 99.95th=[66048],
| 99.99th=[81408]
bw ( KiB/s): min=156481, max=157772, per=100.00%, avg=156978.92, stdev=443.57, samples=13
iops : min=312962, max=315544, avg=313957.92, stdev=887.12, samples=13
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=3.54%, 50=96.31%
lat (usec) : 100=0.13%, 250=0.01%
cpu : usr=15.04%, sys=82.79%, ctx=12432, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2196969,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=1073MiB (1125MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=3157749/0, merge=0/0, ticks=17441/0, in_queue=17441, util=99.09%
$ rm /tmp/fio-output_512_8_io_uring-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751080,
"timestamp_ms" : 1666751080821,
"time" : "Wed Oct 26 04:24:40 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 284537856,
"io_kbytes" : 277869,
"bw_bytes" : 40642459,
"bw" : 39689,
"iops" : 79379.802885,
"runtime" : 7001,
"total_ios" : 555738,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2424,
"max" : 32170,
"mean" : 2650.417359,
"stddev" : 227.123682,
"N" : 555738
},
"clat_ns" : {
"min" : 161,
"max" : 84729,
"mean" : 9278.076117,
"stddev" : 1249.290259,
"N" : 555738,
"percentile" : {
"1.000000" : 8896,
"5.000000" : 8896,
"10.000000" : 9024,
"20.000000" : 9024,
"30.000000" : 9024,
"40.000000" : 9024,
"50.000000" : 9152,
"60.000000" : 9152,
"70.000000" : 9152,
"80.000000" : 9152,
"90.000000" : 9280,
"95.000000" : 9536,
"99.000000" : 15296,
"99.500000" : 15936,
"99.900000" : 19328,
"99.950000" : 27264,
"99.990000" : 55552
}
},
"lat_ns" : {
"min" : 8526,
"max" : 87364,
"mean" : 11928.493475,
"stddev" : 1269.367601,
"N" : 555738
},
"bw_min" : 39567,
"bw_max" : 39849,
"bw_agg" : 100.000000,
"bw_mean" : 39706.571429,
"bw_dev" : 93.082027,
"bw_samples" : 14,
"iops_min" : 79134,
"iops_max" : 79698,
"iops_mean" : 79413.142857,
"iops_stddev" : 186.164054,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.571429,
"sys_cpu" : 49.257143,
"ctx" : 555625,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.871188,
"20" : 3.046580,
"50" : 0.057581,
"100" : 0.016555,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 797598,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 4985,
"write_ticks" : 0,
"in_queue" : 4985,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=139097: Wed Oct 26 04:24:40 2022
read: IOPS=79.4k, BW=38.8MiB/s (40.6MB/s)(271MiB/7001msec)
slat (nsec): min=2424, max=32170, avg=2650.42, stdev=227.12
clat (nsec): min=161, max=84729, avg=9278.08, stdev=1249.29
lat (nsec): min=8526, max=87364, avg=11928.49, stdev=1269.37
clat percentiles (nsec):
| 1.00th=[ 8896], 5.00th=[ 8896], 10.00th=[ 9024], 20.00th=[ 9024],
| 30.00th=[ 9024], 40.00th=[ 9024], 50.00th=[ 9152], 60.00th=[ 9152],
| 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9536],
| 99.00th=[15296], 99.50th=[15936], 99.90th=[19328], 99.95th=[27264],
| 99.99th=[55552]
bw ( KiB/s): min=39567, max=39849, per=100.00%, avg=39706.57, stdev=93.08, samples=14
iops : min=79134, max=79698, avg=79413.14, stdev=186.16, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 10=96.87%, 20=3.05%, 50=0.06%, 100=0.02%
cpu : usr=11.57%, sys=49.26%, ctx=555625, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=555738,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=38.8MiB/s (40.6MB/s), 38.8MiB/s-38.8MiB/s (40.6MB/s-40.6MB/s), io=271MiB (285MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=797598/0, merge=0/0, ticks=4985/0, in_queue=4985, util=99.09%
$ cat /tmp/fio-output_512_8_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752498,
"timestamp_ms" : 1666752498888,
"time" : "Wed Oct 26 04:48:18 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1118396416,
"io_kbytes" : 1092184,
"bw_bytes" : 159748095,
"bw" : 156003,
"iops" : 312006.999000,
"runtime" : 7001,
"total_ios" : 2184361,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1282,
"max" : 29055,
"mean" : 2484.140099,
"stddev" : 984.021916,
"N" : 2184361
},
"clat_ns" : {
"min" : 70,
"max" : 126137,
"mean" : 22762.563038,
"stddev" : 3843.773518,
"N" : 2184368,
"percentile" : {
"1.000000" : 15936,
"5.000000" : 20352,
"10.000000" : 20608,
"20.000000" : 20864,
"30.000000" : 21632,
"40.000000" : 21888,
"50.000000" : 22144,
"60.000000" : 22656,
"70.000000" : 23424,
"80.000000" : 23680,
"90.000000" : 23936,
"95.000000" : 26752,
"99.000000" : 42752,
"99.500000" : 45312,
"99.900000" : 54528,
"99.950000" : 66048,
"99.990000" : 80384
}
},
"lat_ns" : {
"min" : 6782,
"max" : 127519,
"mean" : 25246.704148,
"stddev" : 3767.691495,
"N" : 2184368
},
"bw_min" : 155626,
"bw_max" : 156619,
"bw_agg" : 100.000000,
"bw_mean" : 156072.692308,
"bw_dev" : 295.570461,
"bw_samples" : 13,
"iops_min" : 311252,
"iops_max" : 313238,
"iops_mean" : 312145.692308,
"iops_stddev" : 591.374160,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.014286,
"sys_cpu" : 83.885714,
"ctx" : 12410,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999954,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.028109,
"20" : 3.435879,
"50" : 96.395696,
"100" : 0.135875,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 3135878,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 17469,
"write_ticks" : 0,
"in_queue" : 17469,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=143437: Wed Oct 26 04:48:18 2022
read: IOPS=312k, BW=152MiB/s (160MB/s)(1067MiB/7001msec)
slat (nsec): min=1282, max=29055, avg=2484.14, stdev=984.02
clat (nsec): min=70, max=126137, avg=22762.56, stdev=3843.77
lat (usec): min=6, max=127, avg=25.25, stdev= 3.77
clat percentiles (nsec):
| 1.00th=[15936], 5.00th=[20352], 10.00th=[20608], 20.00th=[20864],
| 30.00th=[21632], 40.00th=[21888], 50.00th=[22144], 60.00th=[22656],
| 70.00th=[23424], 80.00th=[23680], 90.00th=[23936], 95.00th=[26752],
| 99.00th=[42752], 99.50th=[45312], 99.90th=[54528], 99.95th=[66048],
| 99.99th=[80384]
bw ( KiB/s): min=155626, max=156619, per=100.00%, avg=156072.69, stdev=295.57, samples=13
iops : min=311252, max=313238, avg=312145.69, stdev=591.37, samples=13
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=3.44%, 50=96.40%
lat (usec) : 100=0.14%, 250=0.01%
cpu : usr=14.01%, sys=83.89%, ctx=12410, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2184361,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=152MiB/s (160MB/s), 152MiB/s-152MiB/s (160MB/s-160MB/s), io=1067MiB (1118MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=3135878/0, merge=0/0, ticks=17469/0, in_queue=17469, util=99.09%
$ rm /tmp/fio-output_512_8_io_uring-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752512,
"timestamp_ms" : 1666752512505,
"time" : "Wed Oct 26 04:48:32 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1115583488,
"io_kbytes" : 1089437,
"bw_bytes" : 159346305,
"bw" : 155611,
"iops" : 311222.111127,
"runtime" : 7001,
"total_ios" : 2178866,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1292,
"max" : 72896,
"mean" : 2499.882950,
"stddev" : 1010.452035,
"N" : 2178867
},
"clat_ns" : {
"min" : 80,
"max" : 140895,
"mean" : 22828.623283,
"stddev" : 3883.009481,
"N" : 2178873,
"percentile" : {
"1.000000" : 15936,
"5.000000" : 20352,
"10.000000" : 20608,
"20.000000" : 20864,
"30.000000" : 21632,
"40.000000" : 21888,
"50.000000" : 22400,
"60.000000" : 23168,
"70.000000" : 23424,
"80.000000" : 23680,
"90.000000" : 23936,
"95.000000" : 26752,
"99.000000" : 42752,
"99.500000" : 45312,
"99.900000" : 57600,
"99.950000" : 67072,
"99.990000" : 81408
}
},
"lat_ns" : {
"min" : 6842,
"max" : 142227,
"mean" : 25328.506996,
"stddev" : 3810.015386,
"N" : 2178873
},
"bw_min" : 155507,
"bw_max" : 155865,
"bw_agg" : 100.000000,
"bw_mean" : 155698.769231,
"bw_dev" : 108.354475,
"bw_samples" : 13,
"iops_min" : 311014,
"iops_max" : 311730,
"iops_mean" : 311397.538462,
"iops_stddev" : 216.708951,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.385714,
"sys_cpu" : 80.500000,
"ctx" : 12497,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999954,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.010000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.027262,
"20" : 3.408792,
"50" : 96.409784,
"100" : 0.148013,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 3131246,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 17492,
"write_ticks" : 0,
"in_queue" : 17492,
"util" : 99.088748
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=143476: Wed Oct 26 04:48:32 2022
read: IOPS=311k, BW=152MiB/s (159MB/s)(1064MiB/7001msec)
slat (nsec): min=1292, max=72896, avg=2499.88, stdev=1010.45
clat (nsec): min=80, max=140895, avg=22828.62, stdev=3883.01
lat (usec): min=6, max=142, avg=25.33, stdev= 3.81
clat percentiles (nsec):
| 1.00th=[15936], 5.00th=[20352], 10.00th=[20608], 20.00th=[20864],
| 30.00th=[21632], 40.00th=[21888], 50.00th=[22400], 60.00th=[23168],
| 70.00th=[23424], 80.00th=[23680], 90.00th=[23936], 95.00th=[26752],
| 99.00th=[42752], 99.50th=[45312], 99.90th=[57600], 99.95th=[67072],
| 99.99th=[81408]
bw ( KiB/s): min=155507, max=155865, per=100.00%, avg=155698.77, stdev=108.35, samples=13
iops : min=311014, max=311730, avg=311397.54, stdev=216.71, samples=13
lat (nsec) : 100=0.01%, 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=3.41%, 50=96.41%
lat (usec) : 100=0.15%, 250=0.01%
cpu : usr=17.39%, sys=80.50%, ctx=12497, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2178866,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=1064MiB (1116MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=3131246/0, merge=0/0, ticks=17492/0, in_queue=17492, util=99.09%
$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752527,
"timestamp_ms" : 1666752527493,
"time" : "Wed Oct 26 04:48:47 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1236315648,
"io_kbytes" : 1207339,
"bw_bytes" : 176591293,
"bw" : 172452,
"iops" : 344903.870876,
"runtime" : 7001,
"total_ios" : 2414672,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 120,
"max" : 12142,
"mean" : 141.289363,
"stddev" : 45.104120,
"N" : 2414672
},
"clat_ns" : {
"min" : 19116,
"max" : 118853,
"mean" : 21467.040025,
"stddev" : 2785.766655,
"N" : 2414672,
"percentile" : {
"1.000000" : 19584,
"5.000000" : 19840,
"10.000000" : 20096,
"20.000000" : 20096,
"30.000000" : 20352,
"40.000000" : 20608,
"50.000000" : 20864,
"60.000000" : 21120,
"70.000000" : 21376,
"80.000000" : 21632,
"90.000000" : 21888,
"95.000000" : 27520,
"99.000000" : 33536,
"99.500000" : 37632,
"99.900000" : 49920,
"99.950000" : 60672,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 19247,
"max" : 119083,
"mean" : 21608.329388,
"stddev" : 2789.818426,
"N" : 2414672
},
"bw_min" : 172136,
"bw_max" : 172888,
"bw_agg" : 100.000000,
"bw_mean" : 172497.071429,
"bw_dev" : 232.881767,
"bw_samples" : 14,
"iops_min" : 344272,
"iops_max" : 345776,
"iops_mean" : 344994.142857,
"iops_stddev" : 465.763535,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 43.014286,
"sys_cpu" : 56.957143,
"ctx" : 21,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 12.500000,
"2" : 25.000000,
"4" : 50.000000,
"8" : 12.500000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 88.888889,
"8" : 11.111111,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 11.204420,
"50" : 88.695897,
"100" : 0.098978,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=143526: Wed Oct 26 04:48:47 2022
read: IOPS=345k, BW=168MiB/s (177MB/s)(1179MiB/7001msec)
slat (nsec): min=120, max=12142, avg=141.29, stdev=45.10
clat (usec): min=19, max=118, avg=21.47, stdev= 2.79
lat (usec): min=19, max=119, avg=21.61, stdev= 2.79
clat percentiles (nsec):
| 1.00th=[19584], 5.00th=[19840], 10.00th=[20096], 20.00th=[20096],
| 30.00th=[20352], 40.00th=[20608], 50.00th=[20864], 60.00th=[21120],
| 70.00th=[21376], 80.00th=[21632], 90.00th=[21888], 95.00th=[27520],
| 99.00th=[33536], 99.50th=[37632], 99.90th=[49920], 99.95th=[60672],
| 99.99th=[71168]
bw ( KiB/s): min=172136, max=172888, per=100.00%, avg=172497.07, stdev=232.88, samples=14
iops : min=344272, max=345776, avg=344994.14, stdev=465.76, samples=14
lat (usec) : 20=11.20%, 50=88.70%, 100=0.10%, 250=0.01%
cpu : usr=43.01%, sys=56.96%, ctx=21, majf=0, minf=0
IO depths : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=88.9%, 8=11.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2414672,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=168MiB/s (177MB/s), 168MiB/s-168MiB/s (177MB/s-177MB/s), io=1179MiB (1236MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752542,
"timestamp_ms" : 1666752542587,
"time" : "Wed Oct 26 04:49:02 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1230662144,
"io_kbytes" : 1201818,
"bw_bytes" : 175783765,
"bw" : 171663,
"iops" : 343326.667619,
"runtime" : 7001,
"total_ios" : 2403630,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 24797,
"mean" : 132.649914,
"stddev" : 44.296597,
"N" : 2403630
},
"clat_ns" : {
"min" : 16370,
"max" : 105507,
"mean" : 21585.856196,
"stddev" : 2810.338268,
"N" : 2403630,
"percentile" : {
"1.000000" : 19584,
"5.000000" : 19840,
"10.000000" : 20096,
"20.000000" : 20352,
"30.000000" : 20608,
"40.000000" : 20864,
"50.000000" : 21120,
"60.000000" : 21120,
"70.000000" : 21376,
"80.000000" : 21632,
"90.000000" : 22144,
"95.000000" : 27008,
"99.000000" : 35584,
"99.500000" : 38144,
"99.900000" : 51456,
"99.950000" : 61184,
"99.990000" : 71168
}
},
"lat_ns" : {
"min" : 16490,
"max" : 105708,
"mean" : 21718.506110,
"stddev" : 2814.226828,
"N" : 2403630
},
"bw_min" : 171328,
"bw_max" : 172104,
"bw_agg" : 100.000000,
"bw_mean" : 171710.785714,
"bw_dev" : 208.705231,
"bw_samples" : 14,
"iops_min" : 342656,
"iops_max" : 344208,
"iops_mean" : 343421.571429,
"iops_stddev" : 417.410463,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.128571,
"sys_cpu" : 60.842857,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 12.500010,
"2" : 25.000021,
"4" : 50.000000,
"8" : 12.499969,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 88.888881,
"8" : 11.111119,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 7.242213,
"50" : 92.644500,
"100" : 0.112455,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=143575: Wed Oct 26 04:49:02 2022
read: IOPS=343k, BW=168MiB/s (176MB/s)(1174MiB/7001msec)
slat (nsec): min=110, max=24797, avg=132.65, stdev=44.30
clat (usec): min=16, max=105, avg=21.59, stdev= 2.81
lat (usec): min=16, max=105, avg=21.72, stdev= 2.81
clat percentiles (nsec):
| 1.00th=[19584], 5.00th=[19840], 10.00th=[20096], 20.00th=[20352],
| 30.00th=[20608], 40.00th=[20864], 50.00th=[21120], 60.00th=[21120],
| 70.00th=[21376], 80.00th=[21632], 90.00th=[22144], 95.00th=[27008],
| 99.00th=[35584], 99.50th=[38144], 99.90th=[51456], 99.95th=[61184],
| 99.99th=[71168]
bw ( KiB/s): min=171328, max=172104, per=100.00%, avg=171710.79, stdev=208.71, samples=14
iops : min=342656, max=344208, avg=343421.57, stdev=417.41, samples=14
lat (usec) : 20=7.24%, 50=92.64%, 100=0.11%, 250=0.01%
cpu : usr=39.13%, sys=60.84%, ctx=23, majf=0, minf=0
IO depths : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=88.9%, 8=11.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2403630,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s), io=1174MiB (1231MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-reference_2.txt
$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752557,
"timestamp_ms" : 1666752557715,
"time" : "Wed Oct 26 04:49:17 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1238494720,
"io_kbytes" : 1209467,
"bw_bytes" : 176902545,
"bw" : 172756,
"iops" : 345511.784031,
"runtime" : 7001,
"total_ios" : 2418928,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 110,
"max" : 11802,
"mean" : 126.931602,
"stddev" : 38.723132,
"N" : 2418928
},
"clat_ns" : {
"min" : 19076,
"max" : 118472,
"mean" : 21508.297008,
"stddev" : 2755.141185,
"N" : 2418928,
"percentile" : {
"1.000000" : 19584,
"5.000000" : 19840,
"10.000000" : 20096,
"20.000000" : 20352,
"30.000000" : 20608,
"40.000000" : 20864,
"50.000000" : 20864,
"60.000000" : 21120,
"70.000000" : 21376,
"80.000000" : 21632,
"90.000000" : 22144,
"95.000000" : 27008,
"99.000000" : 33536,
"99.500000" : 37632,
"99.900000" : 50432,
"99.950000" : 60672,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 19196,
"max" : 118672,
"mean" : 21635.228610,
"stddev" : 2758.482082,
"N" : 2418928
},
"bw_min" : 172512,
"bw_max" : 173032,
"bw_agg" : 100.000000,
"bw_mean" : 172796.571429,
"bw_dev" : 170.218735,
"bw_samples" : 14,
"iops_min" : 345024,
"iops_max" : 346064,
"iops_mean" : 345593.214286,
"iops_stddev" : 340.424662,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 44.100000,
"sys_cpu" : 55.871429,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 12.500000,
"2" : 25.000000,
"4" : 50.000000,
"8" : 12.500000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 88.888889,
"8" : 11.111111,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 7.970183,
"50" : 91.925431,
"100" : 0.102401,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=143624: Wed Oct 26 04:49:17 2022
read: IOPS=346k, BW=169MiB/s (177MB/s)(1181MiB/7001msec)
slat (nsec): min=110, max=11802, avg=126.93, stdev=38.72
clat (usec): min=19, max=118, avg=21.51, stdev= 2.76
lat (usec): min=19, max=118, avg=21.64, stdev= 2.76
clat percentiles (nsec):
| 1.00th=[19584], 5.00th=[19840], 10.00th=[20096], 20.00th=[20352],
| 30.00th=[20608], 40.00th=[20864], 50.00th=[20864], 60.00th=[21120],
| 70.00th=[21376], 80.00th=[21632], 90.00th=[22144], 95.00th=[27008],
| 99.00th=[33536], 99.50th=[37632], 99.90th=[50432], 99.95th=[60672],
| 99.99th=[72192]
bw ( KiB/s): min=172512, max=173032, per=100.00%, avg=172796.57, stdev=170.22, samples=14
iops : min=345024, max=346064, avg=345593.21, stdev=340.42, samples=14
lat (usec) : 20=7.97%, 50=91.93%, 100=0.10%, 250=0.01%
cpu : usr=44.10%, sys=55.87%, ctx=22, majf=0, minf=0
IO depths : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=88.9%, 8=11.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2418928,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=169MiB/s (177MB/s), 169MiB/s-169MiB/s (177MB/s-177MB/s), io=1181MiB (1238MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752572,
"timestamp_ms" : 1666752572934,
"time" : "Wed Oct 26 04:49:32 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1087555072,
"io_kbytes" : 1062065,
"bw_bytes" : 155342818,
"bw" : 151701,
"iops" : 303402.942437,
"runtime" : 7001,
"total_ios" : 2124124,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1522,
"max" : 29946,
"mean" : 2805.505978,
"stddev" : 1163.675313,
"N" : 2124124
},
"clat_ns" : {
"min" : 190,
"max" : 125075,
"mean" : 23326.719890,
"stddev" : 2999.627077,
"N" : 2124131,
"percentile" : {
"1.000000" : 16768,
"5.000000" : 21376,
"10.000000" : 21632,
"20.000000" : 21888,
"30.000000" : 22656,
"40.000000" : 22912,
"50.000000" : 23168,
"60.000000" : 23168,
"70.000000" : 23680,
"80.000000" : 24192,
"90.000000" : 24448,
"95.000000" : 27264,
"99.000000" : 34560,
"99.500000" : 40192,
"99.900000" : 55040,
"99.950000" : 65280,
"99.990000" : 77312
}
},
"lat_ns" : {
"min" : 6311,
"max" : 126648,
"mean" : 26132.225886,
"stddev" : 2927.082286,
"N" : 2124131
},
"bw_min" : 151465,
"bw_max" : 152209,
"bw_agg" : 100.000000,
"bw_mean" : 151742.642857,
"bw_dev" : 193.711366,
"bw_samples" : 14,
"iops_min" : 302930,
"iops_max" : 304420,
"iops_mean" : 303485.500000,
"iops_stddev" : 388.155550,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 31.928571,
"sys_cpu" : 68.042857,
"ctx" : 26,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999976,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.030177,
"20" : 2.877798,
"50" : 96.939209,
"100" : 0.136903,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143673: Wed Oct 26 04:49:32 2022
read: IOPS=303k, BW=148MiB/s (155MB/s)(1037MiB/7001msec)
slat (nsec): min=1522, max=29946, avg=2805.51, stdev=1163.68
clat (nsec): min=190, max=125075, avg=23326.72, stdev=2999.63
lat (usec): min=6, max=126, avg=26.13, stdev= 2.93
clat percentiles (nsec):
| 1.00th=[16768], 5.00th=[21376], 10.00th=[21632], 20.00th=[21888],
| 30.00th=[22656], 40.00th=[22912], 50.00th=[23168], 60.00th=[23168],
| 70.00th=[23680], 80.00th=[24192], 90.00th=[24448], 95.00th=[27264],
| 99.00th=[34560], 99.50th=[40192], 99.90th=[55040], 99.95th=[65280],
| 99.99th=[77312]
bw ( KiB/s): min=151465, max=152209, per=100.00%, avg=151742.64, stdev=193.71, samples=14
iops : min=302930, max=304420, avg=303485.50, stdev=388.16, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.03%, 20=2.88%, 50=96.94%, 100=0.14%
lat (usec) : 250=0.01%
cpu : usr=31.93%, sys=68.04%, ctx=26, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2124124,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1037MiB (1088MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752588,
"timestamp_ms" : 1666752588001,
"time" : "Wed Oct 26 04:49:48 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1087149056,
"io_kbytes" : 1061669,
"bw_bytes" : 155284824,
"bw" : 151645,
"iops" : 303289.530067,
"runtime" : 7001,
"total_ios" : 2123330,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1523,
"max" : 35066,
"mean" : 2799.298077,
"stddev" : 1164.539197,
"N" : 2123331
},
"clat_ns" : {
"min" : 200,
"max" : 131516,
"mean" : 23335.457212,
"stddev" : 2973.486735,
"N" : 2123337,
"percentile" : {
"1.000000" : 16768,
"5.000000" : 21376,
"10.000000" : 21632,
"20.000000" : 21888,
"30.000000" : 22656,
"40.000000" : 22912,
"50.000000" : 23168,
"60.000000" : 23168,
"70.000000" : 23680,
"80.000000" : 24192,
"90.000000" : 24448,
"95.000000" : 27008,
"99.000000" : 34560,
"99.500000" : 39680,
"99.900000" : 54528,
"99.950000" : 64768,
"99.990000" : 76288
}
},
"lat_ns" : {
"min" : 6322,
"max" : 135254,
"mean" : 26134.754886,
"stddev" : 2903.757258,
"N" : 2123337
},
"bw_min" : 151360,
"bw_max" : 152173,
"bw_agg" : 100.000000,
"bw_mean" : 151693.142857,
"bw_dev" : 212.960761,
"bw_samples" : 14,
"iops_min" : 302720,
"iops_max" : 304346,
"iops_mean" : 303386.500000,
"iops_stddev" : 425.766413,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 30.957143,
"sys_cpu" : 69.014286,
"ctx" : 24,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999976,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.029906,
"20" : 2.873694,
"50" : 96.944752,
"100" : 0.133187,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143723: Wed Oct 26 04:49:47 2022
read: IOPS=303k, BW=148MiB/s (155MB/s)(1037MiB/7001msec)
slat (nsec): min=1523, max=35066, avg=2799.30, stdev=1164.54
clat (nsec): min=200, max=131516, avg=23335.46, stdev=2973.49
lat (usec): min=6, max=135, avg=26.13, stdev= 2.90
clat percentiles (nsec):
| 1.00th=[16768], 5.00th=[21376], 10.00th=[21632], 20.00th=[21888],
| 30.00th=[22656], 40.00th=[22912], 50.00th=[23168], 60.00th=[23168],
| 70.00th=[23680], 80.00th=[24192], 90.00th=[24448], 95.00th=[27008],
| 99.00th=[34560], 99.50th=[39680], 99.90th=[54528], 99.95th=[64768],
| 99.99th=[76288]
bw ( KiB/s): min=151360, max=152173, per=100.00%, avg=151693.14, stdev=212.96, samples=14
iops : min=302720, max=304346, avg=303386.50, stdev=425.77, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.03%, 20=2.87%, 50=96.94%, 100=0.13%
lat (usec) : 250=0.01%
cpu : usr=30.96%, sys=69.01%, ctx=24, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2123330,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1037MiB (1087MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_2.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752603,
"timestamp_ms" : 1666752603063,
"time" : "Wed Oct 26 04:50:03 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1081744896,
"io_kbytes" : 1056391,
"bw_bytes" : 154512911,
"bw" : 150891,
"iops" : 301782.031138,
"runtime" : 7001,
"total_ios" : 2112776,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1522,
"max" : 27091,
"mean" : 2821.886822,
"stddev" : 1220.912718,
"N" : 2112776
},
"clat_ns" : {
"min" : 190,
"max" : 120736,
"mean" : 23444.546176,
"stddev" : 2978.404735,
"N" : 2112783,
"percentile" : {
"1.000000" : 16768,
"5.000000" : 21376,
"10.000000" : 21632,
"20.000000" : 21888,
"30.000000" : 22656,
"40.000000" : 22912,
"50.000000" : 23168,
"60.000000" : 23424,
"70.000000" : 23680,
"80.000000" : 24192,
"90.000000" : 24704,
"95.000000" : 27008,
"99.000000" : 35072,
"99.500000" : 39680,
"99.900000" : 51968,
"99.950000" : 62208,
"99.990000" : 74240
}
},
"lat_ns" : {
"min" : 6622,
"max" : 122389,
"mean" : 26266.430108,
"stddev" : 2931.359656,
"N" : 2112783
},
"bw_min" : 150430,
"bw_max" : 151161,
"bw_agg" : 100.000000,
"bw_mean" : 150901.538462,
"bw_dev" : 191.142013,
"bw_samples" : 13,
"iops_min" : 300860,
"iops_max" : 302322,
"iops_mean" : 301803.076923,
"iops_stddev" : 382.467093,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 31.242857,
"sys_cpu" : 68.728571,
"ctx" : 25,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999976,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.029440,
"20" : 2.844267,
"50" : 96.992298,
"100" : 0.116103,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143773: Wed Oct 26 04:50:03 2022
read: IOPS=302k, BW=147MiB/s (155MB/s)(1032MiB/7001msec)
slat (nsec): min=1522, max=27091, avg=2821.89, stdev=1220.91
clat (nsec): min=190, max=120736, avg=23444.55, stdev=2978.40
lat (usec): min=6, max=122, avg=26.27, stdev= 2.93
clat percentiles (nsec):
| 1.00th=[16768], 5.00th=[21376], 10.00th=[21632], 20.00th=[21888],
| 30.00th=[22656], 40.00th=[22912], 50.00th=[23168], 60.00th=[23424],
| 70.00th=[23680], 80.00th=[24192], 90.00th=[24704], 95.00th=[27008],
| 99.00th=[35072], 99.50th=[39680], 99.90th=[51968], 99.95th=[62208],
| 99.99th=[74240]
bw ( KiB/s): min=150430, max=151161, per=100.00%, avg=150901.54, stdev=191.14, samples=13
iops : min=300860, max=302322, avg=301803.08, stdev=382.47, samples=13
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 4=0.01%, 10=0.03%, 20=2.84%, 50=96.99%, 100=0.12%
lat (usec) : 250=0.01%
cpu : usr=31.24%, sys=68.73%, ctx=25, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2112776,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=147MiB/s (155MB/s), 147MiB/s-147MiB/s (155MB/s-155MB/s), io=1032MiB (1082MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752616,
"timestamp_ms" : 1666752616765,
"time" : "Wed Oct 26 04:50:16 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1090587136,
"io_kbytes" : 1065026,
"bw_bytes" : 155775908,
"bw" : 152124,
"iops" : 304248.678760,
"runtime" : 7001,
"total_ios" : 2130045,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1382,
"max" : 75892,
"mean" : 2807.293279,
"stddev" : 1237.975793,
"N" : 2130046
},
"clat_ns" : {
"min" : 110,
"max" : 140474,
"mean" : 23243.132478,
"stddev" : 4091.818656,
"N" : 2130052,
"percentile" : {
"1.000000" : 15424,
"5.000000" : 19328,
"10.000000" : 19584,
"20.000000" : 21120,
"30.000000" : 21632,
"40.000000" : 22656,
"50.000000" : 22912,
"60.000000" : 23168,
"70.000000" : 23936,
"80.000000" : 24704,
"90.000000" : 25984,
"95.000000" : 27264,
"99.000000" : 43264,
"99.500000" : 45824,
"99.900000" : 56576,
"99.950000" : 66048,
"99.990000" : 81408
}
},
"lat_ns" : {
"min" : 6923,
"max" : 141916,
"mean" : 26050.426103,
"stddev" : 4239.486761,
"N" : 2130052
},
"bw_min" : 151886,
"bw_max" : 152476,
"bw_agg" : 100.000000,
"bw_mean" : 152167.357143,
"bw_dev" : 193.514304,
"bw_samples" : 14,
"iops_min" : 303772,
"iops_max" : 304952,
"iops_mean" : 304334.714286,
"iops_stddev" : 386.864800,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 17.885714,
"sys_cpu" : 80.342857,
"ctx" : 11485,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999977,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.044882,
"20" : 15.144750,
"50" : 84.647883,
"100" : 0.147649,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=143812: Wed Oct 26 04:50:16 2022
read: IOPS=304k, BW=149MiB/s (156MB/s)(1040MiB/7001msec)
slat (nsec): min=1382, max=75892, avg=2807.29, stdev=1237.98
clat (nsec): min=110, max=140474, avg=23243.13, stdev=4091.82
lat (usec): min=6, max=141, avg=26.05, stdev= 4.24
clat percentiles (nsec):
| 1.00th=[15424], 5.00th=[19328], 10.00th=[19584], 20.00th=[21120],
| 30.00th=[21632], 40.00th=[22656], 50.00th=[22912], 60.00th=[23168],
| 70.00th=[23936], 80.00th=[24704], 90.00th=[25984], 95.00th=[27264],
| 99.00th=[43264], 99.50th=[45824], 99.90th=[56576], 99.95th=[66048],
| 99.99th=[81408]
bw ( KiB/s): min=151886, max=152476, per=100.00%, avg=152167.36, stdev=193.51, samples=14
iops : min=303772, max=304952, avg=304334.71, stdev=386.86, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.04%, 20=15.14%, 50=84.65%
lat (usec) : 100=0.15%, 250=0.01%
cpu : usr=17.89%, sys=80.34%, ctx=11485, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2130045,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=1040MiB (1091MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752630,
"timestamp_ms" : 1666752630403,
"time" : "Wed Oct 26 04:50:30 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1091710976,
"io_kbytes" : 1066124,
"bw_bytes" : 155936434,
"bw" : 152281,
"iops" : 304562.205399,
"runtime" : 7001,
"total_ios" : 2132240,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1372,
"max" : 65753,
"mean" : 2806.080467,
"stddev" : 1233.481209,
"N" : 2132241
},
"clat_ns" : {
"min" : 110,
"max" : 142047,
"mean" : 23217.680802,
"stddev" : 4093.965853,
"N" : 2132247,
"percentile" : {
"1.000000" : 15424,
"5.000000" : 19328,
"10.000000" : 19584,
"20.000000" : 21120,
"30.000000" : 21632,
"40.000000" : 22656,
"50.000000" : 22912,
"60.000000" : 23168,
"70.000000" : 23936,
"80.000000" : 24704,
"90.000000" : 25984,
"95.000000" : 27264,
"99.000000" : 43264,
"99.500000" : 45824,
"99.900000" : 57600,
"99.950000" : 67072,
"99.990000" : 81408
}
},
"lat_ns" : {
"min" : 6883,
"max" : 143439,
"mean" : 26023.761436,
"stddev" : 4231.145211,
"N" : 2132247
},
"bw_min" : 152213,
"bw_max" : 152540,
"bw_agg" : 100.000000,
"bw_mean" : 152322.357143,
"bw_dev" : 94.171372,
"bw_samples" : 14,
"iops_min" : 304426,
"iops_max" : 305080,
"iops_mean" : 304644.857143,
"iops_stddev" : 188.408254,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.142857,
"sys_cpu" : 79.042857,
"ctx" : 11717,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999977,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.046196,
"20" : 14.669643,
"50" : 85.119499,
"100" : 0.149655,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=143851: Wed Oct 26 04:50:30 2022
read: IOPS=305k, BW=149MiB/s (156MB/s)(1041MiB/7001msec)
slat (nsec): min=1372, max=65753, avg=2806.08, stdev=1233.48
clat (nsec): min=110, max=142047, avg=23217.68, stdev=4093.97
lat (usec): min=6, max=143, avg=26.02, stdev= 4.23
clat percentiles (nsec):
| 1.00th=[15424], 5.00th=[19328], 10.00th=[19584], 20.00th=[21120],
| 30.00th=[21632], 40.00th=[22656], 50.00th=[22912], 60.00th=[23168],
| 70.00th=[23936], 80.00th=[24704], 90.00th=[25984], 95.00th=[27264],
| 99.00th=[43264], 99.50th=[45824], 99.90th=[57600], 99.95th=[67072],
| 99.99th=[81408]
bw ( KiB/s): min=152213, max=152540, per=100.00%, avg=152322.36, stdev=94.17, samples=14
iops : min=304426, max=305080, avg=304644.86, stdev=188.41, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.05%, 20=14.67%, 50=85.12%
lat (usec) : 100=0.15%, 250=0.01%
cpu : usr=19.14%, sys=79.04%, ctx=11717, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2132240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=1041MiB (1092MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_io_uring-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751094,
"timestamp_ms" : 1666751094444,
"time" : "Wed Oct 26 04:24:54 2022",
"jobs" : [
{
"jobname" : "io_uring-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "io_uring",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 286309376,
"io_kbytes" : 279599,
"bw_bytes" : 40895497,
"bw" : 39937,
"iops" : 79874.017997,
"runtime" : 7001,
"total_ios" : 559198,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2494,
"max" : 45485,
"mean" : 2642.445052,
"stddev" : 228.167226,
"N" : 559198
},
"clat_ns" : {
"min" : 190,
"max" : 81473,
"mean" : 9223.417267,
"stddev" : 1276.987473,
"N" : 559198,
"percentile" : {
"1.000000" : 8768,
"5.000000" : 8896,
"10.000000" : 8896,
"20.000000" : 9024,
"30.000000" : 9024,
"40.000000" : 9024,
"50.000000" : 9024,
"60.000000" : 9024,
"70.000000" : 9152,
"80.000000" : 9152,
"90.000000" : 9152,
"95.000000" : 9408,
"99.000000" : 15424,
"99.500000" : 16192,
"99.900000" : 19584,
"99.950000" : 23168,
"99.990000" : 56576
}
},
"lat_ns" : {
"min" : 8957,
"max" : 84108,
"mean" : 11865.862319,
"stddev" : 1296.542780,
"N" : 559198
},
"bw_min" : 39762,
"bw_max" : 40171,
"bw_agg" : 100.000000,
"bw_mean" : 39954.785714,
"bw_dev" : 110.705554,
"bw_samples" : 14,
"iops_min" : 79524,
"iops_max" : 80342,
"iops_mean" : 79909.642857,
"iops_stddev" : 221.379454,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.328571,
"sys_cpu" : 46.328571,
"ctx" : 559054,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 97.150741,
"20" : 2.758057,
"50" : 0.062590,
"100" : 0.018062,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 801785,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5027,
"write_ticks" : 0,
"in_queue" : 5027,
"util" : 99.098564
}
]
}
io_uring-reference: (groupid=0, jobs=1): err= 0: pid=139136: Wed Oct 26 04:24:54 2022
read: IOPS=79.9k, BW=39.0MiB/s (40.9MB/s)(273MiB/7001msec)
slat (nsec): min=2494, max=45485, avg=2642.45, stdev=228.17
clat (nsec): min=190, max=81473, avg=9223.42, stdev=1276.99
lat (nsec): min=8957, max=84108, avg=11865.86, stdev=1296.54
clat percentiles (nsec):
| 1.00th=[ 8768], 5.00th=[ 8896], 10.00th=[ 8896], 20.00th=[ 9024],
| 30.00th=[ 9024], 40.00th=[ 9024], 50.00th=[ 9024], 60.00th=[ 9024],
| 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9152], 95.00th=[ 9408],
| 99.00th=[15424], 99.50th=[16192], 99.90th=[19584], 99.95th=[23168],
| 99.99th=[56576]
bw ( KiB/s): min=39762, max=40171, per=100.00%, avg=39954.79, stdev=110.71, samples=14
iops : min=79524, max=80342, avg=79909.64, stdev=221.38, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 10=97.15%, 20=2.76%, 50=0.06%, 100=0.02%
cpu : usr=14.33%, sys=46.33%, ctx=559054, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=559198,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=39.0MiB/s (40.9MB/s), 39.0MiB/s-39.0MiB/s (40.9MB/s-40.9MB/s), io=273MiB (286MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=801785/0, merge=0/0, ticks=5027/0, in_queue=5027, util=99.10%
$ cat /tmp/fio-output_512_8_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752644,
"timestamp_ms" : 1666752644045,
"time" : "Wed Oct 26 04:50:44 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1097063424,
"io_kbytes" : 1071351,
"bw_bytes" : 156700960,
"bw" : 153028,
"iops" : 306055.420654,
"runtime" : 7001,
"total_ios" : 2142694,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1362,
"max" : 43081,
"mean" : 2770.524205,
"stddev" : 1257.270481,
"N" : 2142695
},
"clat_ns" : {
"min" : 110,
"max" : 142568,
"mean" : 23118.608042,
"stddev" : 3988.582641,
"N" : 2142701,
"percentile" : {
"1.000000" : 15424,
"5.000000" : 19328,
"10.000000" : 20864,
"20.000000" : 21120,
"30.000000" : 21376,
"40.000000" : 22400,
"50.000000" : 22656,
"60.000000" : 22912,
"70.000000" : 23680,
"80.000000" : 24192,
"90.000000" : 25472,
"95.000000" : 27776,
"99.000000" : 42752,
"99.500000" : 45824,
"99.900000" : 58624,
"99.950000" : 68096,
"99.990000" : 82432
}
},
"lat_ns" : {
"min" : 6923,
"max" : 143950,
"mean" : 25889.131866,
"stddev" : 4007.357932,
"N" : 2142701
},
"bw_min" : 152809,
"bw_max" : 153283,
"bw_agg" : 100.000000,
"bw_mean" : 153068.142857,
"bw_dev" : 168.665740,
"bw_samples" : 14,
"iops_min" : 305618,
"iops_max" : 306566,
"iops_mean" : 306136.214286,
"iops_stddev" : 337.609738,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 19.928571,
"sys_cpu" : 78.200000,
"ctx" : 12004,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999977,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 0.040043,
"20" : 8.114318,
"50" : 91.677533,
"100" : 0.156905,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=143892: Wed Oct 26 04:50:44 2022
read: IOPS=306k, BW=149MiB/s (157MB/s)(1046MiB/7001msec)
slat (nsec): min=1362, max=43081, avg=2770.52, stdev=1257.27
clat (nsec): min=110, max=142568, avg=23118.61, stdev=3988.58
lat (usec): min=6, max=143, avg=25.89, stdev= 4.01
clat percentiles (nsec):
| 1.00th=[15424], 5.00th=[19328], 10.00th=[20864], 20.00th=[21120],
| 30.00th=[21376], 40.00th=[22400], 50.00th=[22656], 60.00th=[22912],
| 70.00th=[23680], 80.00th=[24192], 90.00th=[25472], 95.00th=[27776],
| 99.00th=[42752], 99.50th=[45824], 99.90th=[58624], 99.95th=[68096],
| 99.99th=[82432]
bw ( KiB/s): min=152809, max=153283, per=100.00%, avg=153068.14, stdev=168.67, samples=14
iops : min=305618, max=306566, avg=306136.21, stdev=337.61, samples=14
lat (nsec) : 250=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.04%, 20=8.11%, 50=91.68%
lat (usec) : 100=0.16%, 250=0.01%
cpu : usr=19.93%, sys=78.20%, ctx=12004, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2142694,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=149MiB/s (157MB/s), 149MiB/s-149MiB/s (157MB/s-157MB/s), io=1046MiB (1097MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-reference_0.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752657,
"timestamp_ms" : 1666752657713,
"time" : "Wed Oct 26 04:50:57 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1046272512,
"io_kbytes" : 1021750,
"bw_bytes" : 149446152,
"bw" : 145943,
"iops" : 291885.873447,
"runtime" : 7001,
"total_ios" : 2043493,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1402,
"max" : 113964,
"mean" : 2707.785157,
"stddev" : 1337.154939,
"N" : 2043494
},
"clat_ns" : {
"min" : 2595,
"max" : 138590,
"mean" : 24451.180669,
"stddev" : 3177.840339,
"N" : 2043500,
"percentile" : {
"1.000000" : 17536,
"5.000000" : 22400,
"10.000000" : 22656,
"20.000000" : 22656,
"30.000000" : 22912,
"40.000000" : 23680,
"50.000000" : 24192,
"60.000000" : 24448,
"70.000000" : 25216,
"80.000000" : 25472,
"90.000000" : 25728,
"95.000000" : 29312,
"99.000000" : 37632,
"99.500000" : 42240,
"99.900000" : 54528,
"99.950000" : 67072,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 10450,
"max" : 143930,
"mean" : 27158.965725,
"stddev" : 3022.018116,
"N" : 2043500
},
"bw_min" : 145819,
"bw_max" : 146239,
"bw_agg" : 100.000000,
"bw_mean" : 146001.692308,
"bw_dev" : 141.640028,
"bw_samples" : 13,
"iops_min" : 291638,
"iops_max" : 292478,
"iops_mean" : 292003.461538,
"iops_stddev" : 283.176216,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 35.971429,
"sys_cpu" : 62.771429,
"ctx" : 10182,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999951,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 2.033870,
"50" : 97.828033,
"100" : 0.127282,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2858731,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 15931,
"write_ticks" : 0,
"in_queue" : 15931,
"util" : 99.035239
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=143932: Wed Oct 26 04:50:57 2022
read: IOPS=292k, BW=143MiB/s (149MB/s)(998MiB/7001msec)
slat (nsec): min=1402, max=113964, avg=2707.79, stdev=1337.15
clat (usec): min=2, max=138, avg=24.45, stdev= 3.18
lat (usec): min=10, max=143, avg=27.16, stdev= 3.02
clat percentiles (nsec):
| 1.00th=[17536], 5.00th=[22400], 10.00th=[22656], 20.00th=[22656],
| 30.00th=[22912], 40.00th=[23680], 50.00th=[24192], 60.00th=[24448],
| 70.00th=[25216], 80.00th=[25472], 90.00th=[25728], 95.00th=[29312],
| 99.00th=[37632], 99.50th=[42240], 99.90th=[54528], 99.95th=[67072],
| 99.99th=[79360]
bw ( KiB/s): min=145819, max=146239, per=100.00%, avg=146001.69, stdev=141.64, samples=13
iops : min=291638, max=292478, avg=292003.46, stdev=283.18, samples=13
lat (usec) : 4=0.01%, 10=0.01%, 20=2.03%, 50=97.83%, 100=0.13%
lat (usec) : 250=0.01%
cpu : usr=35.97%, sys=62.77%, ctx=10182, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2043493,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=143MiB/s (149MB/s), 143MiB/s-143MiB/s (149MB/s-149MB/s), io=998MiB (1046MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2858731/0, merge=0/0, ticks=15931/0, in_queue=15931, util=99.04%
$ rm /tmp/fio-output_512_8_libaio-reference_1.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752671,
"timestamp_ms" : 1666752671349,
"time" : "Wed Oct 26 04:51:11 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1046752768,
"io_kbytes" : 1022219,
"bw_bytes" : 149514750,
"bw" : 146010,
"iops" : 292019.854307,
"runtime" : 7001,
"total_ios" : 2044431,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1403,
"max" : 123251,
"mean" : 2715.633511,
"stddev" : 1330.313775,
"N" : 2044432
},
"clat_ns" : {
"min" : 2605,
"max" : 145643,
"mean" : 24432.144750,
"stddev" : 3169.438036,
"N" : 2044438,
"percentile" : {
"1.000000" : 17536,
"5.000000" : 22400,
"10.000000" : 22656,
"20.000000" : 22656,
"30.000000" : 22912,
"40.000000" : 23680,
"50.000000" : 23936,
"60.000000" : 24448,
"70.000000" : 24960,
"80.000000" : 25472,
"90.000000" : 25728,
"95.000000" : 29312,
"99.000000" : 37632,
"99.500000" : 41728,
"99.900000" : 52992,
"99.950000" : 65280,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 10781,
"max" : 148709,
"mean" : 27147.777613,
"stddev" : 3031.587372,
"N" : 2044438
},
"bw_min" : 145798,
"bw_max" : 146766,
"bw_agg" : 100.000000,
"bw_mean" : 146077.000000,
"bw_dev" : 239.693394,
"bw_samples" : 14,
"iops_min" : 291596,
"iops_max" : 293532,
"iops_mean" : 292154.142857,
"iops_stddev" : 479.286466,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 38.614286,
"sys_cpu" : 60.157143,
"ctx" : 10093,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999951,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 0.010000,
"20" : 2.060720,
"50" : 97.807507,
"100" : 0.119887,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2933310,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 16349,
"write_ticks" : 0,
"in_queue" : 16349,
"util" : 99.098564
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=143970: Wed Oct 26 04:51:11 2022
read: IOPS=292k, BW=143MiB/s (150MB/s)(998MiB/7001msec)
slat (nsec): min=1403, max=123251, avg=2715.63, stdev=1330.31
clat (usec): min=2, max=145, avg=24.43, stdev= 3.17
lat (usec): min=10, max=148, avg=27.15, stdev= 3.03
clat percentiles (nsec):
| 1.00th=[17536], 5.00th=[22400], 10.00th=[22656], 20.00th=[22656],
| 30.00th=[22912], 40.00th=[23680], 50.00th=[23936], 60.00th=[24448],
| 70.00th=[24960], 80.00th=[25472], 90.00th=[25728], 95.00th=[29312],
| 99.00th=[37632], 99.50th=[41728], 99.90th=[52992], 99.95th=[65280],
| 99.99th=[79360]
bw ( KiB/s): min=145798, max=146766, per=100.00%, avg=146077.00, stdev=239.69, samples=14
iops : min=291596, max=293532, avg=292154.14, stdev=479.29, samples=14
lat (usec) : 4=0.01%, 10=0.01%, 20=2.06%, 50=97.81%, 100=0.12%
lat (usec) : 250=0.01%
cpu : usr=38.61%, sys=60.16%, ctx=10093, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2044431,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=998MiB (1047MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2933310/0, merge=0/0, ticks=16349/0, in_queue=16349, util=99.10%
$ rm /tmp/fio-output_512_8_libaio-reference_2.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752685,
"timestamp_ms" : 1666752685000,
"time" : "Wed Oct 26 04:51:25 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1043872256,
"io_kbytes" : 1019406,
"bw_bytes" : 149103307,
"bw" : 145608,
"iops" : 291216.254821,
"runtime" : 7001,
"total_ios" : 2038805,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1432,
"max" : 27141,
"mean" : 2732.441553,
"stddev" : 1335.007083,
"N" : 2038806
},
"clat_ns" : {
"min" : 4679,
"max" : 159368,
"mean" : 24492.867077,
"stddev" : 3138.428213,
"N" : 2038812,
"percentile" : {
"1.000000" : 17536,
"5.000000" : 22400,
"10.000000" : 22656,
"20.000000" : 22912,
"30.000000" : 22912,
"40.000000" : 23680,
"50.000000" : 24192,
"60.000000" : 24960,
"70.000000" : 25216,
"80.000000" : 25472,
"90.000000" : 25728,
"95.000000" : 29312,
"99.000000" : 37632,
"99.500000" : 41728,
"99.900000" : 52480,
"99.950000" : 65280,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 10911,
"max" : 160841,
"mean" : 27225.311592,
"stddev" : 2968.109528,
"N" : 2038812
},
"bw_min" : 145239,
"bw_max" : 145957,
"bw_agg" : 100.000000,
"bw_mean" : 145648.230769,
"bw_dev" : 185.987972,
"bw_samples" : 13,
"iops_min" : 290478,
"iops_max" : 291914,
"iops_mean" : 291296.538462,
"iops_stddev" : 372.038218,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 37.700000,
"sys_cpu" : 61.085714,
"ctx" : 10154,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999951,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 1.971253,
"50" : 97.900584,
"100" : 0.117912,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 2855695,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 15886,
"write_ticks" : 0,
"in_queue" : 15886,
"util" : 99.045298
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=144008: Wed Oct 26 04:51:24 2022
read: IOPS=291k, BW=142MiB/s (149MB/s)(996MiB/7001msec)
slat (nsec): min=1432, max=27141, avg=2732.44, stdev=1335.01
clat (usec): min=4, max=159, avg=24.49, stdev= 3.14
lat (usec): min=10, max=160, avg=27.23, stdev= 2.97
clat percentiles (nsec):
| 1.00th=[17536], 5.00th=[22400], 10.00th=[22656], 20.00th=[22912],
| 30.00th=[22912], 40.00th=[23680], 50.00th=[24192], 60.00th=[24960],
| 70.00th=[25216], 80.00th=[25472], 90.00th=[25728], 95.00th=[29312],
| 99.00th=[37632], 99.50th=[41728], 99.90th=[52480], 99.95th=[65280],
| 99.99th=[78336]
bw ( KiB/s): min=145239, max=145957, per=100.00%, avg=145648.23, stdev=185.99, samples=13
iops : min=290478, max=291914, avg=291296.54, stdev=372.04, samples=13
lat (usec) : 10=0.01%, 20=1.97%, 50=97.90%, 100=0.12%, 250=0.01%
cpu : usr=37.70%, sys=61.09%, ctx=10154, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2038805,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=996MiB (1044MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=2855695/0, merge=0/0, ticks=15886/0, in_queue=15886, util=99.05%
$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_0.txt
$ rm /tmp/fio-output_512_8_libaio-bdev_aio_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752700,
"timestamp_ms" : 1666752700018,
"time" : "Wed Oct 26 04:51:40 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1086418944,
"io_kbytes" : 1060956,
"bw_bytes" : 155180537,
"bw" : 151543,
"iops" : 303086.559063,
"runtime" : 7001,
"total_ios" : 2121909,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1593,
"max" : 35116,
"mean" : 2769.186245,
"stddev" : 1002.279645,
"N" : 2121910
},
"clat_ns" : {
"min" : 330,
"max" : 188353,
"mean" : 14078.489138,
"stddev" : 6683.083761,
"N" : 2121912,
"percentile" : {
"1.000000" : 3408,
"5.000000" : 3536,
"10.000000" : 4832,
"20.000000" : 6816,
"30.000000" : 9664,
"40.000000" : 12096,
"50.000000" : 14272,
"60.000000" : 16320,
"70.000000" : 18816,
"80.000000" : 20608,
"90.000000" : 22912,
"95.000000" : 23424,
"99.000000" : 27264,
"99.500000" : 29824,
"99.900000" : 42752,
"99.950000" : 57088,
"99.990000" : 70144
}
},
"lat_ns" : {
"min" : 6182,
"max" : 191369,
"mean" : 16847.674926,
"stddev" : 6639.843683,
"N" : 2121912
},
"bw_min" : 150945,
"bw_max" : 152439,
"bw_agg" : 100.000000,
"bw_mean" : 151590.285714,
"bw_dev" : 485.280320,
"bw_samples" : 14,
"iops_min" : 301890,
"iops_max" : 304878,
"iops_mean" : 303180.571429,
"iops_stddev" : 970.850675,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 40.557143,
"sys_cpu" : 59.414286,
"ctx" : 26,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.471274,
"2" : 26.285576,
"4" : 58.295997,
"8" : 14.947154,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 87.443970,
"8" : 12.556030,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.033743,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 6.563524,
"10" : 29.667955,
"20" : 42.265856,
"50" : 21.394131,
"100" : 0.072718,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=144056: Wed Oct 26 04:51:40 2022
read: IOPS=303k, BW=148MiB/s (155MB/s)(1036MiB/7001msec)
slat (nsec): min=1593, max=35116, avg=2769.19, stdev=1002.28
clat (nsec): min=330, max=188353, avg=14078.49, stdev=6683.08
lat (usec): min=6, max=191, avg=16.85, stdev= 6.64
clat percentiles (nsec):
| 1.00th=[ 3408], 5.00th=[ 3536], 10.00th=[ 4832], 20.00th=[ 6816],
| 30.00th=[ 9664], 40.00th=[12096], 50.00th=[14272], 60.00th=[16320],
| 70.00th=[18816], 80.00th=[20608], 90.00th=[22912], 95.00th=[23424],
| 99.00th=[27264], 99.50th=[29824], 99.90th=[42752], 99.95th=[57088],
| 99.99th=[70144]
bw ( KiB/s): min=150945, max=152439, per=100.00%, avg=151590.29, stdev=485.28, samples=14
iops : min=301890, max=304878, avg=303180.57, stdev=970.85, samples=14
lat (nsec) : 500=0.03%, 750=0.01%
lat (usec) : 2=0.01%, 4=6.56%, 10=29.67%, 20=42.27%, 50=21.39%
lat (usec) : 100=0.07%, 250=0.01%
cpu : usr=40.56%, sys=59.41%, ctx=26, majf=0, minf=0
IO depths : 1=0.5%, 2=26.3%, 4=58.3%, 8=14.9%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=87.4%, 8=12.6%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2121909,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1036MiB (1086MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_aio_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752715,
"timestamp_ms" : 1666752715116,
"time" : "Wed Oct 26 04:51:55 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1076485120,
"io_kbytes" : 1051255,
"bw_bytes" : 153761622,
"bw" : 150157,
"iops" : 300314.669333,
"runtime" : 7001,
"total_ios" : 2102503,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1593,
"max" : 36158,
"mean" : 2832.892318,
"stddev" : 1021.358270,
"N" : 2102503
},
"clat_ns" : {
"min" : 350,
"max" : 128501,
"mean" : 14078.870878,
"stddev" : 6722.192899,
"N" : 2102503,
"percentile" : {
"1.000000" : 3440,
"5.000000" : 3568,
"10.000000" : 4832,
"20.000000" : 6816,
"30.000000" : 9664,
"40.000000" : 11712,
"50.000000" : 14144,
"60.000000" : 16320,
"70.000000" : 18304,
"80.000000" : 19840,
"90.000000" : 23168,
"95.000000" : 23680,
"99.000000" : 27008,
"99.500000" : 29824,
"99.900000" : 44800,
"99.950000" : 57600,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 6161,
"max" : 130174,
"mean" : 16911.763196,
"stddev" : 6736.191451,
"N" : 2102503
},
"bw_min" : 149226,
"bw_max" : 150989,
"bw_agg" : 100.000000,
"bw_mean" : 150194.785714,
"bw_dev" : 508.754464,
"bw_samples" : 14,
"iops_min" : 298452,
"iops_max" : 301979,
"iops_mean" : 300389.642857,
"iops_stddev" : 1017.629040,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 40.442857,
"sys_cpu" : 59.528571,
"ctx" : 29,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.466064,
"2" : 26.752542,
"4" : 57.926671,
"8" : 14.854723,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 87.516984,
"8" : 12.483016,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.042806,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 6.502393,
"10" : 29.740600,
"20" : 43.938059,
"50" : 19.695097,
"100" : 0.078906,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=144105: Wed Oct 26 04:51:55 2022
read: IOPS=300k, BW=147MiB/s (154MB/s)(1027MiB/7001msec)
slat (nsec): min=1593, max=36158, avg=2832.89, stdev=1021.36
clat (nsec): min=350, max=128501, avg=14078.87, stdev=6722.19
lat (usec): min=6, max=130, avg=16.91, stdev= 6.74
clat percentiles (nsec):
| 1.00th=[ 3440], 5.00th=[ 3568], 10.00th=[ 4832], 20.00th=[ 6816],
| 30.00th=[ 9664], 40.00th=[11712], 50.00th=[14144], 60.00th=[16320],
| 70.00th=[18304], 80.00th=[19840], 90.00th=[23168], 95.00th=[23680],
| 99.00th=[27008], 99.50th=[29824], 99.90th=[44800], 99.95th=[57600],
| 99.99th=[69120]
bw ( KiB/s): min=149226, max=150989, per=100.00%, avg=150194.79, stdev=508.75, samples=14
iops : min=298452, max=301979, avg=300389.64, stdev=1017.63, samples=14
lat (nsec) : 500=0.04%, 750=0.01%
lat (usec) : 2=0.01%, 4=6.50%, 10=29.74%, 20=43.94%, 50=19.70%
lat (usec) : 100=0.08%, 250=0.01%
cpu : usr=40.44%, sys=59.53%, ctx=29, majf=0, minf=0
IO depths : 1=0.5%, 2=26.8%, 4=57.9%, 8=14.9%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2102503,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=1027MiB (1076MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_aio_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752730,
"timestamp_ms" : 1666752730123,
"time" : "Wed Oct 26 04:52:10 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1091307008,
"io_kbytes" : 1065729,
"bw_bytes" : 155878732,
"bw" : 152225,
"iops" : 304450.078560,
"runtime" : 7001,
"total_ios" : 2131455,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1572,
"max" : 80341,
"mean" : 2723.375660,
"stddev" : 926.510920,
"N" : 2131456
},
"clat_ns" : {
"min" : 320,
"max" : 145703,
"mean" : 14031.642536,
"stddev" : 6669.051166,
"N" : 2131459,
"percentile" : {
"1.000000" : 3376,
"5.000000" : 3504,
"10.000000" : 4832,
"20.000000" : 6688,
"30.000000" : 9536,
"40.000000" : 11968,
"50.000000" : 14144,
"60.000000" : 16512,
"70.000000" : 18560,
"80.000000" : 20352,
"90.000000" : 22912,
"95.000000" : 23424,
"99.000000" : 27008,
"99.500000" : 29312,
"99.900000" : 45312,
"99.950000" : 58624,
"99.990000" : 70144
}
},
"lat_ns" : {
"min" : 6122,
"max" : 147336,
"mean" : 16755.017869,
"stddev" : 6618.416459,
"N" : 2131459
},
"bw_min" : 151543,
"bw_max" : 152997,
"bw_agg" : 100.000000,
"bw_mean" : 152264.642857,
"bw_dev" : 377.366300,
"bw_samples" : 14,
"iops_min" : 303086,
"iops_max" : 305994,
"iops_mean" : 304529.357143,
"iops_stddev" : 754.613463,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 41.871429,
"sys_cpu" : 58.100000,
"ctx" : 26,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.501066,
"2" : 26.540462,
"4" : 58.078777,
"8" : 14.879695,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 87.483879,
"8" : 12.516121,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.044148,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 6.314982,
"10" : 29.727393,
"20" : 43.166475,
"50" : 20.663397,
"100" : 0.081963,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=144155: Wed Oct 26 04:52:10 2022
read: IOPS=304k, BW=149MiB/s (156MB/s)(1041MiB/7001msec)
slat (nsec): min=1572, max=80341, avg=2723.38, stdev=926.51
clat (nsec): min=320, max=145703, avg=14031.64, stdev=6669.05
lat (usec): min=6, max=147, avg=16.76, stdev= 6.62
clat percentiles (nsec):
| 1.00th=[ 3376], 5.00th=[ 3504], 10.00th=[ 4832], 20.00th=[ 6688],
| 30.00th=[ 9536], 40.00th=[11968], 50.00th=[14144], 60.00th=[16512],
| 70.00th=[18560], 80.00th=[20352], 90.00th=[22912], 95.00th=[23424],
| 99.00th=[27008], 99.50th=[29312], 99.90th=[45312], 99.95th=[58624],
| 99.99th=[70144]
bw ( KiB/s): min=151543, max=152997, per=100.00%, avg=152264.64, stdev=377.37, samples=14
iops : min=303086, max=305994, avg=304529.36, stdev=754.61, samples=14
lat (nsec) : 500=0.04%, 750=0.01%
lat (usec) : 2=0.01%, 4=6.31%, 10=29.73%, 20=43.17%, 50=20.66%
lat (usec) : 100=0.08%, 250=0.01%
cpu : usr=41.87%, sys=58.10%, ctx=26, majf=0, minf=0
IO depths : 1=0.5%, 2=26.5%, 4=58.1%, 8=14.9%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2131455,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=1041MiB (1091MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752745,
"timestamp_ms" : 1666752745825,
"time" : "Wed Oct 26 04:52:25 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1052019200,
"io_kbytes" : 1027362,
"bw_bytes" : 150266990,
"bw" : 146745,
"iops" : 293489.072990,
"runtime" : 7001,
"total_ios" : 2054717,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1623,
"max" : 44804,
"mean" : 2930.171240,
"stddev" : 1093.681472,
"N" : 2054718
},
"clat_ns" : {
"min" : 751,
"max" : 124714,
"mean" : 14597.804246,
"stddev" : 6757.196878,
"N" : 2054717,
"percentile" : {
"1.000000" : 3920,
"5.000000" : 4048,
"10.000000" : 5280,
"20.000000" : 8256,
"30.000000" : 10176,
"40.000000" : 12096,
"50.000000" : 14656,
"60.000000" : 16768,
"70.000000" : 18560,
"80.000000" : 20096,
"90.000000" : 24192,
"95.000000" : 24448,
"99.000000" : 27264,
"99.500000" : 30336,
"99.900000" : 42752,
"99.950000" : 56576,
"99.990000" : 69120
}
},
"lat_ns" : {
"min" : 6472,
"max" : 126397,
"mean" : 17527.975420,
"stddev" : 6799.693481,
"N" : 2054717
},
"bw_min" : 146440,
"bw_max" : 147365,
"bw_agg" : 100.000000,
"bw_mean" : 146789.214286,
"bw_dev" : 284.407935,
"bw_samples" : 14,
"iops_min" : 292880,
"iops_max" : 294730,
"iops_mean" : 293578.500000,
"iops_stddev" : 568.809925,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.014286,
"sys_cpu" : 60.957143,
"ctx" : 27,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.407453,
"2" : 26.810310,
"4" : 57.959417,
"8" : 14.822820,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 87.512790,
"8" : 12.487210,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.029931
},
"latency_us" : {
"2" : 0.010000,
"4" : 3.778185,
"10" : 23.005261,
"20" : 50.434439,
"50" : 22.678257,
"100" : 0.072613,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=144204: Wed Oct 26 04:52:25 2022
read: IOPS=293k, BW=143MiB/s (150MB/s)(1003MiB/7001msec)
slat (nsec): min=1623, max=44804, avg=2930.17, stdev=1093.68
clat (nsec): min=751, max=124714, avg=14597.80, stdev=6757.20
lat (usec): min=6, max=126, avg=17.53, stdev= 6.80
clat percentiles (nsec):
| 1.00th=[ 3920], 5.00th=[ 4048], 10.00th=[ 5280], 20.00th=[ 8256],
| 30.00th=[10176], 40.00th=[12096], 50.00th=[14656], 60.00th=[16768],
| 70.00th=[18560], 80.00th=[20096], 90.00th=[24192], 95.00th=[24448],
| 99.00th=[27264], 99.50th=[30336], 99.90th=[42752], 99.95th=[56576],
| 99.99th=[69120]
bw ( KiB/s): min=146440, max=147365, per=100.00%, avg=146789.21, stdev=284.41, samples=14
iops : min=292880, max=294730, avg=293578.50, stdev=568.81, samples=14
lat (nsec) : 1000=0.03%
lat (usec) : 2=0.01%, 4=3.78%, 10=23.01%, 20=50.43%, 50=22.68%
lat (usec) : 100=0.07%, 250=0.01%
cpu : usr=39.01%, sys=60.96%, ctx=27, majf=0, minf=0
IO depths : 1=0.4%, 2=26.8%, 4=58.0%, 8=14.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2054717,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=1003MiB (1052MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752761,
"timestamp_ms" : 1666752761025,
"time" : "Wed Oct 26 04:52:41 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1073317888,
"io_kbytes" : 1048162,
"bw_bytes" : 153309225,
"bw" : 149716,
"iops" : 299431.366948,
"runtime" : 7001,
"total_ios" : 2096319,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1593,
"max" : 80861,
"mean" : 2771.613320,
"stddev" : 938.858044,
"N" : 2096320
},
"clat_ns" : {
"min" : 732,
"max" : 127409,
"mean" : 14376.397846,
"stddev" : 6697.479279,
"N" : 2096324,
"percentile" : {
"1.000000" : 3792,
"5.000000" : 3888,
"10.000000" : 5152,
"20.000000" : 7200,
"30.000000" : 10048,
"40.000000" : 11840,
"50.000000" : 14528,
"60.000000" : 16768,
"70.000000" : 18304,
"80.000000" : 19840,
"90.000000" : 23680,
"95.000000" : 24192,
"99.000000" : 27264,
"99.500000" : 29824,
"99.900000" : 41728,
"99.950000" : 56576,
"99.990000" : 70144
}
},
"lat_ns" : {
"min" : 6502,
"max" : 129052,
"mean" : 17148.011537,
"stddev" : 6676.946354,
"N" : 2096324
},
"bw_min" : 149307,
"bw_max" : 150245,
"bw_agg" : 100.000000,
"bw_mean" : 149750.857143,
"bw_dev" : 298.817891,
"bw_samples" : 14,
"iops_min" : 298614,
"iops_max" : 300490,
"iops_mean" : 299501.714286,
"iops_stddev" : 597.243338,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.985714,
"sys_cpu" : 59.985714,
"ctx" : 25,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.448739,
"2" : 26.840428,
"4" : 57.885036,
"8" : 14.825797,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 87.522870,
"8" : 12.477130,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.039975
},
"latency_us" : {
"2" : 0.010000,
"4" : 6.707996,
"10" : 23.033327,
"20" : 51.503469,
"50" : 18.641581,
"100" : 0.071936,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=144253: Wed Oct 26 04:52:41 2022
read: IOPS=299k, BW=146MiB/s (153MB/s)(1024MiB/7001msec)
slat (nsec): min=1593, max=80861, avg=2771.61, stdev=938.86
clat (nsec): min=732, max=127409, avg=14376.40, stdev=6697.48
lat (usec): min=6, max=129, avg=17.15, stdev= 6.68
clat percentiles (nsec):
| 1.00th=[ 3792], 5.00th=[ 3888], 10.00th=[ 5152], 20.00th=[ 7200],
| 30.00th=[10048], 40.00th=[11840], 50.00th=[14528], 60.00th=[16768],
| 70.00th=[18304], 80.00th=[19840], 90.00th=[23680], 95.00th=[24192],
| 99.00th=[27264], 99.50th=[29824], 99.90th=[41728], 99.95th=[56576],
| 99.99th=[70144]
bw ( KiB/s): min=149307, max=150245, per=100.00%, avg=149750.86, stdev=298.82, samples=14
iops : min=298614, max=300490, avg=299501.71, stdev=597.24, samples=14
lat (nsec) : 750=0.01%, 1000=0.04%
lat (usec) : 2=0.01%, 4=6.71%, 10=23.03%, 20=51.50%, 50=18.64%
lat (usec) : 100=0.07%, 250=0.01%
cpu : usr=39.99%, sys=59.99%, ctx=25, majf=0, minf=0
IO depths : 1=0.4%, 2=26.8%, 4=57.9%, 8=14.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2096319,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1024MiB (1073MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752776,
"timestamp_ms" : 1666752776202,
"time" : "Wed Oct 26 04:52:56 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1060783104,
"io_kbytes" : 1035921,
"bw_bytes" : 151518797,
"bw" : 147967,
"iops" : 295934.152264,
"runtime" : 7001,
"total_ios" : 2071835,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1623,
"max" : 49042,
"mean" : 2860.583254,
"stddev" : 1041.203387,
"N" : 2071836
},
"clat_ns" : {
"min" : 762,
"max" : 127149,
"mean" : 14498.881318,
"stddev" : 6774.396456,
"N" : 2071842,
"percentile" : {
"1.000000" : 3856,
"5.000000" : 3984,
"10.000000" : 5216,
"20.000000" : 8256,
"30.000000" : 10176,
"40.000000" : 11968,
"50.000000" : 14528,
"60.000000" : 17280,
"70.000000" : 18560,
"80.000000" : 19840,
"90.000000" : 23936,
"95.000000" : 24448,
"99.000000" : 27520,
"99.500000" : 30592,
"99.900000" : 46336,
"99.950000" : 58624,
"99.990000" : 72192
}
},
"lat_ns" : {
"min" : 6472,
"max" : 128822,
"mean" : 17359.464180,
"stddev" : 6757.687788,
"N" : 2071842
},
"bw_min" : 147584,
"bw_max" : 148654,
"bw_agg" : 100.000000,
"bw_mean" : 148008.500000,
"bw_dev" : 324.912710,
"bw_samples" : 14,
"iops_min" : 295168,
"iops_max" : 297308,
"iops_mean" : 296017.000000,
"iops_stddev" : 650.166251,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.242857,
"sys_cpu" : 60.728571,
"ctx" : 24,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.416249,
"2" : 26.862419,
"4" : 57.902777,
"8" : 14.818555,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 87.526588,
"8" : 12.473412,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.035331
},
"latency_us" : {
"2" : 0.010000,
"4" : 5.719471,
"10" : 21.807528,
"20" : 52.966718,
"50" : 19.386679,
"100" : 0.083260,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=144302: Wed Oct 26 04:52:56 2022
read: IOPS=296k, BW=144MiB/s (152MB/s)(1012MiB/7001msec)
slat (nsec): min=1623, max=49042, avg=2860.58, stdev=1041.20
clat (nsec): min=762, max=127149, avg=14498.88, stdev=6774.40
lat (usec): min=6, max=128, avg=17.36, stdev= 6.76
clat percentiles (nsec):
| 1.00th=[ 3856], 5.00th=[ 3984], 10.00th=[ 5216], 20.00th=[ 8256],
| 30.00th=[10176], 40.00th=[11968], 50.00th=[14528], 60.00th=[17280],
| 70.00th=[18560], 80.00th=[19840], 90.00th=[23936], 95.00th=[24448],
| 99.00th=[27520], 99.50th=[30592], 99.90th=[46336], 99.95th=[58624],
| 99.99th=[72192]
bw ( KiB/s): min=147584, max=148654, per=100.00%, avg=148008.50, stdev=324.91, samples=14
iops : min=295168, max=297308, avg=296017.00, stdev=650.17, samples=14
lat (nsec) : 1000=0.04%
lat (usec) : 2=0.01%, 4=5.72%, 10=21.81%, 20=52.97%, 50=19.39%
lat (usec) : 100=0.08%, 250=0.01%
cpu : usr=39.24%, sys=60.73%, ctx=24, majf=0, minf=0
IO depths : 1=0.4%, 2=26.9%, 4=57.9%, 8=14.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2071835,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=144MiB/s (152MB/s), 144MiB/s-144MiB/s (152MB/s-152MB/s), io=1012MiB (1061MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751109,
"timestamp_ms" : 1666751109430,
"time" : "Wed Oct 26 04:25:09 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 554295808,
"io_kbytes" : 541304,
"bw_bytes" : 79173804,
"bw" : 77318,
"iops" : 154636.337666,
"runtime" : 7001,
"total_ios" : 1082609,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 160,
"max" : 11602,
"mean" : 177.716369,
"stddev" : 44.536553,
"N" : 1082609
},
"clat_ns" : {
"min" : 5681,
"max" : 127730,
"mean" : 6024.580633,
"stddev" : 1112.814196,
"N" : 1082609,
"percentile" : {
"1.000000" : 5728,
"5.000000" : 5792,
"10.000000" : 5792,
"20.000000" : 5792,
"30.000000" : 5792,
"40.000000" : 5856,
"50.000000" : 5856,
"60.000000" : 5920,
"70.000000" : 5920,
"80.000000" : 5920,
"90.000000" : 6048,
"95.000000" : 6112,
"99.000000" : 11328,
"99.500000" : 12608,
"99.900000" : 17024,
"99.950000" : 17024,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 5851,
"max" : 127900,
"mean" : 6202.297002,
"stddev" : 1114.031677,
"N" : 1082609
},
"bw_min" : 77106,
"bw_max" : 77614,
"bw_agg" : 100.000000,
"bw_mean" : 77336.642857,
"bw_dev" : 146.503141,
"bw_samples" : 14,
"iops_min" : 154212,
"iops_max" : 155228,
"iops_mean" : 154673.285714,
"iops_stddev" : 293.006282,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 51.814286,
"sys_cpu" : 48.157143,
"ctx" : 21,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.603836,
"20" : 1.381108,
"50" : 0.010000,
"100" : 0.012378,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=139184: Wed Oct 26 04:25:09 2022
read: IOPS=155k, BW=75.5MiB/s (79.2MB/s)(529MiB/7001msec)
slat (nsec): min=160, max=11602, avg=177.72, stdev=44.54
clat (usec): min=5, max=127, avg= 6.02, stdev= 1.11
lat (usec): min=5, max=127, avg= 6.20, stdev= 1.11
clat percentiles (nsec):
| 1.00th=[ 5728], 5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5792],
| 30.00th=[ 5792], 40.00th=[ 5856], 50.00th=[ 5856], 60.00th=[ 5920],
| 70.00th=[ 5920], 80.00th=[ 5920], 90.00th=[ 6048], 95.00th=[ 6112],
| 99.00th=[11328], 99.50th=[12608], 99.90th=[17024], 99.95th=[17024],
| 99.99th=[58112]
bw ( KiB/s): min=77106, max=77614, per=100.00%, avg=77336.64, stdev=146.50, samples=14
iops : min=154212, max=155228, avg=154673.29, stdev=293.01, samples=14
lat (usec) : 10=98.60%, 20=1.38%, 50=0.01%, 100=0.01%, 250=0.01%
cpu : usr=51.81%, sys=48.16%, ctx=21, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1082609,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=75.5MiB/s (79.2MB/s), 75.5MiB/s-75.5MiB/s (79.2MB/s-79.2MB/s), io=529MiB (554MB), run=7001-7001msec
$ cat /tmp/fio-output_512_8_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752789,
"timestamp_ms" : 1666752789979,
"time" : "Wed Oct 26 04:53:09 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1014642688,
"io_kbytes" : 990862,
"bw_bytes" : 144928251,
"bw" : 141531,
"iops" : 283061.848307,
"runtime" : 7001,
"total_ios" : 1981716,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1472,
"max" : 32762,
"mean" : 2358.838814,
"stddev" : 1033.707278,
"N" : 1981717
},
"clat_ns" : {
"min" : 6823,
"max" : 153127,
"mean" : 25221.540123,
"stddev" : 3166.343719,
"N" : 1981723,
"percentile" : {
"1.000000" : 18304,
"5.000000" : 23168,
"10.000000" : 23424,
"20.000000" : 23424,
"30.000000" : 23680,
"40.000000" : 23680,
"50.000000" : 24960,
"60.000000" : 25984,
"70.000000" : 26240,
"80.000000" : 26240,
"90.000000" : 26496,
"95.000000" : 29568,
"99.000000" : 38144,
"99.500000" : 42752,
"99.900000" : 51968,
"99.950000" : 64768,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 10910,
"max" : 154660,
"mean" : 27580.380426,
"stddev" : 2971.138538,
"N" : 1981723
},
"bw_min" : 141148,
"bw_max" : 141868,
"bw_agg" : 100.000000,
"bw_mean" : 141569.500000,
"bw_dev" : 202.955755,
"bw_samples" : 14,
"iops_min" : 282296,
"iops_max" : 283736,
"iops_mean" : 283139.142857,
"iops_stddev" : 405.859175,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 36.985714,
"sys_cpu" : 62.000000,
"ctx" : 9204,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999975,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 1.889827,
"50" : 97.980891,
"100" : 0.126507,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=144341: Wed Oct 26 04:53:09 2022
read: IOPS=283k, BW=138MiB/s (145MB/s)(968MiB/7001msec)
slat (nsec): min=1472, max=32762, avg=2358.84, stdev=1033.71
clat (usec): min=6, max=153, avg=25.22, stdev= 3.17
lat (usec): min=10, max=154, avg=27.58, stdev= 2.97
clat percentiles (nsec):
| 1.00th=[18304], 5.00th=[23168], 10.00th=[23424], 20.00th=[23424],
| 30.00th=[23680], 40.00th=[23680], 50.00th=[24960], 60.00th=[25984],
| 70.00th=[26240], 80.00th=[26240], 90.00th=[26496], 95.00th=[29568],
| 99.00th=[38144], 99.50th=[42752], 99.90th=[51968], 99.95th=[64768],
| 99.99th=[78336]
bw ( KiB/s): min=141148, max=141868, per=100.00%, avg=141569.50, stdev=202.96, samples=14
iops : min=282296, max=283736, avg=283139.14, stdev=405.86, samples=14
lat (usec) : 10=0.01%, 20=1.89%, 50=97.98%, 100=0.13%, 250=0.01%
cpu : usr=36.99%, sys=62.00%, ctx=9204, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1981716,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=968MiB (1015MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752803,
"timestamp_ms" : 1666752803651,
"time" : "Wed Oct 26 04:53:23 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1012346368,
"io_kbytes" : 988619,
"bw_bytes" : 144600252,
"bw" : 141211,
"iops" : 282421.368376,
"runtime" : 7001,
"total_ios" : 1977232,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1482,
"max" : 54362,
"mean" : 2372.692545,
"stddev" : 1039.385184,
"N" : 1977232
},
"clat_ns" : {
"min" : 4679,
"max" : 146735,
"mean" : 25274.840964,
"stddev" : 3228.285563,
"N" : 1977239,
"percentile" : {
"1.000000" : 18304,
"5.000000" : 23168,
"10.000000" : 23424,
"20.000000" : 23424,
"30.000000" : 23680,
"40.000000" : 23936,
"50.000000" : 24960,
"60.000000" : 25984,
"70.000000" : 26240,
"80.000000" : 26240,
"90.000000" : 26496,
"95.000000" : 30080,
"99.000000" : 38656,
"99.500000" : 42752,
"99.900000" : 53504,
"99.950000" : 66048,
"99.990000" : 79360
}
},
"lat_ns" : {
"min" : 6211,
"max" : 151143,
"mean" : 27647.531768,
"stddev" : 3048.297465,
"N" : 1977239
},
"bw_min" : 140956,
"bw_max" : 141621,
"bw_agg" : 100.000000,
"bw_mean" : 141248.214286,
"bw_dev" : 159.472293,
"bw_samples" : 14,
"iops_min" : 281912,
"iops_max" : 283242,
"iops_mean" : 282496.500000,
"iops_stddev" : 318.744351,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 38.000000,
"sys_cpu" : 60.957143,
"ctx" : 9034,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999975,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 1.892090,
"50" : 97.959976,
"100" : 0.144242,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=144380: Wed Oct 26 04:53:23 2022
read: IOPS=282k, BW=138MiB/s (145MB/s)(965MiB/7001msec)
slat (nsec): min=1482, max=54362, avg=2372.69, stdev=1039.39
clat (usec): min=4, max=146, avg=25.27, stdev= 3.23
lat (usec): min=6, max=151, avg=27.65, stdev= 3.05
clat percentiles (nsec):
| 1.00th=[18304], 5.00th=[23168], 10.00th=[23424], 20.00th=[23424],
| 30.00th=[23680], 40.00th=[23936], 50.00th=[24960], 60.00th=[25984],
| 70.00th=[26240], 80.00th=[26240], 90.00th=[26496], 95.00th=[30080],
| 99.00th=[38656], 99.50th=[42752], 99.90th=[53504], 99.95th=[66048],
| 99.99th=[79360]
bw ( KiB/s): min=140956, max=141621, per=100.00%, avg=141248.21, stdev=159.47, samples=14
iops : min=281912, max=283242, avg=282496.50, stdev=318.74, samples=14
lat (usec) : 10=0.01%, 20=1.89%, 50=97.96%, 100=0.14%, 250=0.01%
cpu : usr=38.00%, sys=60.96%, ctx=9034, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1977232,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=965MiB (1012MB), run=7001-7001msec
$ rm /tmp/fio-output_512_8_libaio-xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666752817,
"timestamp_ms" : 1666752817296,
"time" : "Wed Oct 26 04:53:37 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "8",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 1011662336,
"io_kbytes" : 987951,
"bw_bytes" : 144502547,
"bw" : 141115,
"iops" : 282230.395658,
"runtime" : 7001,
"total_ios" : 1975895,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1482,
"max" : 25537,
"mean" : 2298.559704,
"stddev" : 931.379939,
"N" : 1975896
},
"clat_ns" : {
"min" : 4278,
"max" : 139502,
"mean" : 25292.891398,
"stddev" : 3159.053006,
"N" : 1975902,
"percentile" : {
"1.000000" : 18304,
"5.000000" : 23168,
"10.000000" : 23424,
"20.000000" : 23680,
"30.000000" : 23680,
"40.000000" : 23936,
"50.000000" : 24960,
"60.000000" : 25984,
"70.000000" : 26240,
"80.000000" : 26496,
"90.000000" : 26496,
"95.000000" : 29824,
"99.000000" : 38144,
"99.500000" : 42752,
"99.900000" : 52480,
"99.950000" : 66048,
"99.990000" : 78336
}
},
"lat_ns" : {
"min" : 5811,
"max" : 141034,
"mean" : 27591.452594,
"stddev" : 2976.337309,
"N" : 1975902
},
"bw_min" : 140763,
"bw_max" : 141655,
"bw_agg" : 100.000000,
"bw_mean" : 141148.214286,
"bw_dev" : 223.325281,
"bw_samples" : 14,
"iops_min" : 281526,
"iops_max" : 283310,
"iops_mean" : 282296.428571,
"iops_stddev" : 446.650563,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 39.814286,
"sys_cpu" : 59.185714,
"ctx" : 8781,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 0.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 100.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 99.999975,
"8" : 0.100000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.010000,
"20" : 1.898886,
"50" : 97.964619,
"100" : 0.134015,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 8,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=144420: Wed Oct 26 04:53:37 2022
read: IOPS=282k, BW=138MiB/s (145MB/s)(965MiB/7001msec)
slat (nsec): min=1482, max=25537, avg=2298.56, stdev=931.38
clat (usec): min=4, max=139, avg=25.29, stdev= 3.16
lat (usec): min=5, max=141, avg=27.59, stdev= 2.98
clat percentiles (nsec):
| 1.00th=[18304], 5.00th=[23168], 10.00th=[23424], 20.00th=[23680],
| 30.00th=[23680], 40.00th=[23936], 50.00th=[24960], 60.00th=[25984],
| 70.00th=[26240], 80.00th=[26496], 90.00th=[26496], 95.00th=[29824],
| 99.00th=[38144], 99.50th=[42752], 99.90th=[52480], 99.95th=[66048],
| 99.99th=[78336]
bw ( KiB/s): min=140763, max=141655, per=100.00%, avg=141148.21, stdev=223.33, samples=14
iops : min=281526, max=283310, avg=282296.43, stdev=446.65, samples=14
lat (usec) : 10=0.01%, 20=1.90%, 50=97.96%, 100=0.13%, 250=0.01%
cpu : usr=39.81%, sys=59.19%, ctx=8781, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1975895,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=965MiB (1012MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751124,
"timestamp_ms" : 1666751124416,
"time" : "Wed Oct 26 04:25:24 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 554997760,
"io_kbytes" : 541990,
"bw_bytes" : 79285394,
"bw" : 77427,
"iops" : 154854.285714,
"runtime" : 7000,
"total_ios" : 1083980,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 160,
"max" : 12163,
"mean" : 182.966398,
"stddev" : 38.361146,
"N" : 1083980
},
"clat_ns" : {
"min" : 5690,
"max" : 79259,
"mean" : 6045.423559,
"stddev" : 1095.010971,
"N" : 1083980,
"percentile" : {
"1.000000" : 5792,
"5.000000" : 5792,
"10.000000" : 5792,
"20.000000" : 5856,
"30.000000" : 5856,
"40.000000" : 5856,
"50.000000" : 5920,
"60.000000" : 5920,
"70.000000" : 5920,
"80.000000" : 5984,
"90.000000" : 6048,
"95.000000" : 6176,
"99.000000" : 11456,
"99.500000" : 12096,
"99.900000" : 17024,
"99.950000" : 17024,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 5871,
"max" : 79429,
"mean" : 6228.389957,
"stddev" : 1095.955735,
"N" : 1083980
},
"bw_min" : 77263,
"bw_max" : 77571,
"bw_agg" : 100.000000,
"bw_mean" : 77436.857143,
"bw_dev" : 79.987362,
"bw_samples" : 14,
"iops_min" : 154526,
"iops_max" : 155142,
"iops_mean" : 154873.785714,
"iops_stddev" : 160.033736,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 48.900000,
"sys_cpu" : 51.071429,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.581247,
"20" : 1.403808,
"50" : 0.010000,
"100" : 0.013100,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=139233: Wed Oct 26 04:25:24 2022
read: IOPS=155k, BW=75.6MiB/s (79.3MB/s)(529MiB/7000msec)
slat (nsec): min=160, max=12163, avg=182.97, stdev=38.36
clat (nsec): min=5690, max=79259, avg=6045.42, stdev=1095.01
lat (nsec): min=5871, max=79429, avg=6228.39, stdev=1095.96
clat percentiles (nsec):
| 1.00th=[ 5792], 5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5856],
| 30.00th=[ 5856], 40.00th=[ 5856], 50.00th=[ 5920], 60.00th=[ 5920],
| 70.00th=[ 5920], 80.00th=[ 5984], 90.00th=[ 6048], 95.00th=[ 6176],
| 99.00th=[11456], 99.50th=[12096], 99.90th=[17024], 99.95th=[17024],
| 99.99th=[58112]
bw ( KiB/s): min=77263, max=77571, per=100.00%, avg=77436.86, stdev=79.99, samples=14
iops : min=154526, max=155142, avg=154873.79, stdev=160.03, samples=14
lat (usec) : 10=98.58%, 20=1.40%, 50=0.01%, 100=0.01%
cpu : usr=48.90%, sys=51.07%, ctx=23, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1083980,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=75.6MiB/s (79.3MB/s), 75.6MiB/s-75.6MiB/s (79.3MB/s-79.3MB/s), io=529MiB (555MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751139,
"timestamp_ms" : 1666751139437,
"time" : "Wed Oct 26 04:25:39 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_uring",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_uring",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 556163072,
"io_kbytes" : 543128,
"bw_bytes" : 79440518,
"bw" : 77578,
"iops" : 155157.263248,
"runtime" : 7001,
"total_ios" : 1086256,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 160,
"max" : 11702,
"mean" : 182.538771,
"stddev" : 44.114911,
"N" : 1086256
},
"clat_ns" : {
"min" : 5681,
"max" : 114124,
"mean" : 6027.226244,
"stddev" : 1126.443798,
"N" : 1086256,
"percentile" : {
"1.000000" : 5728,
"5.000000" : 5792,
"10.000000" : 5792,
"20.000000" : 5792,
"30.000000" : 5792,
"40.000000" : 5856,
"50.000000" : 5856,
"60.000000" : 5920,
"70.000000" : 5920,
"80.000000" : 5920,
"90.000000" : 6048,
"95.000000" : 6112,
"99.000000" : 11456,
"99.500000" : 12736,
"99.900000" : 17024,
"99.950000" : 17024,
"99.990000" : 58624
}
},
"lat_ns" : {
"min" : 5861,
"max" : 114305,
"mean" : 6209.765015,
"stddev" : 1127.684770,
"N" : 1086256
},
"bw_min" : 77442,
"bw_max" : 77945,
"bw_agg" : 100.000000,
"bw_mean" : 77599.142857,
"bw_dev" : 130.531374,
"bw_samples" : 14,
"iops_min" : 154884,
"iops_max" : 155890,
"iops_mean" : 155198.357143,
"iops_stddev" : 261.172388,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 47.071429,
"sys_cpu" : 52.900000,
"ctx" : 18,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 98.564611,
"20" : 1.419371,
"50" : 0.010000,
"100" : 0.013809,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=139283: Wed Oct 26 04:25:39 2022
read: IOPS=155k, BW=75.8MiB/s (79.4MB/s)(530MiB/7001msec)
slat (nsec): min=160, max=11702, avg=182.54, stdev=44.11
clat (usec): min=5, max=114, avg= 6.03, stdev= 1.13
lat (usec): min=5, max=114, avg= 6.21, stdev= 1.13
clat percentiles (nsec):
| 1.00th=[ 5728], 5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5792],
| 30.00th=[ 5792], 40.00th=[ 5856], 50.00th=[ 5856], 60.00th=[ 5920],
| 70.00th=[ 5920], 80.00th=[ 5920], 90.00th=[ 6048], 95.00th=[ 6112],
| 99.00th=[11456], 99.50th=[12736], 99.90th=[17024], 99.95th=[17024],
| 99.99th=[58624]
bw ( KiB/s): min=77442, max=77945, per=100.00%, avg=77599.14, stdev=130.53, samples=14
iops : min=154884, max=155890, avg=155198.36, stdev=261.17, samples=14
lat (usec) : 10=98.56%, 20=1.42%, 50=0.01%, 100=0.01%, 250=0.01%
cpu : usr=47.07%, sys=52.90%, ctx=18, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1086256,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=75.8MiB/s (79.4MB/s), 75.8MiB/s-75.8MiB/s (79.4MB/s-79.4MB/s), io=530MiB (556MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751154,
"timestamp_ms" : 1666751154404,
"time" : "Wed Oct 26 04:25:54 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 579297280,
"io_kbytes" : 565720,
"bw_bytes" : 82744933,
"bw" : 80805,
"iops" : 161611.055563,
"runtime" : 7001,
"total_ios" : 1131439,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1573,
"max" : 22613,
"mean" : 1629.937999,
"stddev" : 110.064556,
"N" : 1131440
},
"clat_ns" : {
"min" : 231,
"max" : 100949,
"mean" : 4328.043267,
"stddev" : 1152.033943,
"N" : 1131439,
"percentile" : {
"1.000000" : 4048,
"5.000000" : 4080,
"10.000000" : 4080,
"20.000000" : 4128,
"30.000000" : 4128,
"40.000000" : 4128,
"50.000000" : 4192,
"60.000000" : 4192,
"70.000000" : 4256,
"80.000000" : 4256,
"90.000000" : 4320,
"95.000000" : 4448,
"99.000000" : 10048,
"99.500000" : 10688,
"99.900000" : 15424,
"99.950000" : 15552,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5620,
"max" : 102592,
"mean" : 5957.981281,
"stddev" : 1158.044008,
"N" : 1131439
},
"bw_min" : 80689,
"bw_max" : 80965,
"bw_agg" : 100.000000,
"bw_mean" : 80824.285714,
"bw_dev" : 84.986360,
"bw_samples" : 14,
"iops_min" : 161378,
"iops_max" : 161930,
"iops_mean" : 161648.642857,
"iops_stddev" : 169.934651,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.557143,
"sys_cpu" : 89.414286,
"ctx" : 24,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.957080,
"20" : 1.019763,
"50" : 0.010000,
"100" : 0.014495,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139333: Wed Oct 26 04:25:54 2022
read: IOPS=162k, BW=78.9MiB/s (82.7MB/s)(552MiB/7001msec)
slat (nsec): min=1573, max=22613, avg=1629.94, stdev=110.06
clat (nsec): min=231, max=100949, avg=4328.04, stdev=1152.03
lat (usec): min=5, max=102, avg= 5.96, stdev= 1.16
clat percentiles (nsec):
| 1.00th=[ 4048], 5.00th=[ 4080], 10.00th=[ 4080], 20.00th=[ 4128],
| 30.00th=[ 4128], 40.00th=[ 4128], 50.00th=[ 4192], 60.00th=[ 4192],
| 70.00th=[ 4256], 80.00th=[ 4256], 90.00th=[ 4320], 95.00th=[ 4448],
| 99.00th=[10048], 99.50th=[10688], 99.90th=[15424], 99.95th=[15552],
| 99.99th=[57088]
bw ( KiB/s): min=80689, max=80965, per=100.00%, avg=80824.29, stdev=84.99, samples=14
iops : min=161378, max=161930, avg=161648.64, stdev=169.93, samples=14
lat (nsec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=98.96%, 20=1.02%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=10.56%, sys=89.41%, ctx=24, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1131439,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=78.9MiB/s (82.7MB/s), 78.9MiB/s-78.9MiB/s (82.7MB/s-82.7MB/s), io=552MiB (579MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751169,
"timestamp_ms" : 1666751169479,
"time" : "Wed Oct 26 04:26:09 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 578703872,
"io_kbytes" : 565140,
"bw_bytes" : 82671981,
"bw" : 80734,
"iops" : 161468.714286,
"runtime" : 7000,
"total_ios" : 1130281,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1563,
"max" : 15979,
"mean" : 1627.234484,
"stddev" : 111.403997,
"N" : 1130281
},
"clat_ns" : {
"min" : 230,
"max" : 75993,
"mean" : 4333.185862,
"stddev" : 1142.444879,
"N" : 1130281,
"percentile" : {
"1.000000" : 4080,
"5.000000" : 4080,
"10.000000" : 4128,
"20.000000" : 4128,
"30.000000" : 4128,
"40.000000" : 4192,
"50.000000" : 4192,
"60.000000" : 4192,
"70.000000" : 4256,
"80.000000" : 4256,
"90.000000" : 4320,
"95.000000" : 4448,
"99.000000" : 10048,
"99.500000" : 11200,
"99.900000" : 15424,
"99.950000" : 15552,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5630,
"max" : 77626,
"mean" : 5960.420346,
"stddev" : 1148.508228,
"N" : 1130281
},
"bw_min" : 80670,
"bw_max" : 80870,
"bw_agg" : 100.000000,
"bw_mean" : 80741.785714,
"bw_dev" : 54.574125,
"bw_samples" : 14,
"iops_min" : 161340,
"iops_max" : 161740,
"iops_mean" : 161483.571429,
"iops_stddev" : 109.148251,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 11.014286,
"sys_cpu" : 88.957143,
"ctx" : 20,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.965390,
"20" : 1.013199,
"50" : 0.010000,
"100" : 0.012740,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139382: Wed Oct 26 04:26:09 2022
read: IOPS=161k, BW=78.8MiB/s (82.7MB/s)(552MiB/7000msec)
slat (nsec): min=1563, max=15979, avg=1627.23, stdev=111.40
clat (nsec): min=230, max=75993, avg=4333.19, stdev=1142.44
lat (nsec): min=5630, max=77626, avg=5960.42, stdev=1148.51
clat percentiles (nsec):
| 1.00th=[ 4080], 5.00th=[ 4080], 10.00th=[ 4128], 20.00th=[ 4128],
| 30.00th=[ 4128], 40.00th=[ 4192], 50.00th=[ 4192], 60.00th=[ 4192],
| 70.00th=[ 4256], 80.00th=[ 4256], 90.00th=[ 4320], 95.00th=[ 4448],
| 99.00th=[10048], 99.50th=[11200], 99.90th=[15424], 99.95th=[15552],
| 99.99th=[57088]
bw ( KiB/s): min=80670, max=80870, per=100.00%, avg=80741.79, stdev=54.57, samples=14
iops : min=161340, max=161740, avg=161483.57, stdev=109.15, samples=14
lat (nsec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=98.97%, 20=1.01%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=11.01%, sys=88.96%, ctx=20, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1130281,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=78.8MiB/s (82.7MB/s), 78.8MiB/s-78.8MiB/s (82.7MB/s-82.7MB/s), io=552MiB (579MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751184,
"timestamp_ms" : 1666751184500,
"time" : "Wed Oct 26 04:26:24 2022",
"jobs" : [
{
"jobname" : "io_uring-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 576747520,
"io_kbytes" : 563230,
"bw_bytes" : 82380734,
"bw" : 80449,
"iops" : 160899.871447,
"runtime" : 7001,
"total_ios" : 1126460,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1563,
"max" : 15619,
"mean" : 1623.509082,
"stddev" : 104.723698,
"N" : 1126460
},
"clat_ns" : {
"min" : 241,
"max" : 77606,
"mean" : 4359.351258,
"stddev" : 1122.867630,
"N" : 1126460,
"percentile" : {
"1.000000" : 4128,
"5.000000" : 4128,
"10.000000" : 4128,
"20.000000" : 4128,
"30.000000" : 4128,
"40.000000" : 4192,
"50.000000" : 4192,
"60.000000" : 4256,
"70.000000" : 4256,
"80.000000" : 4320,
"90.000000" : 4384,
"95.000000" : 4448,
"99.000000" : 10048,
"99.500000" : 10688,
"99.900000" : 15424,
"99.950000" : 15552,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5650,
"max" : 79209,
"mean" : 5982.860339,
"stddev" : 1128.826196,
"N" : 1126460
},
"bw_min" : 80316,
"bw_max" : 80630,
"bw_agg" : 100.000000,
"bw_mean" : 80486.615385,
"bw_dev" : 102.524419,
"bw_samples" : 13,
"iops_min" : 160632,
"iops_max" : 161260,
"iops_mean" : 160973.230769,
"iops_stddev" : 205.048837,
"iops_samples" : 13
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 10.328571,
"sys_cpu" : 89.642857,
"ctx" : 21,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.971468,
"20" : 1.008291,
"50" : 0.010000,
"100" : 0.013050,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139431: Wed Oct 26 04:26:24 2022
read: IOPS=161k, BW=78.6MiB/s (82.4MB/s)(550MiB/7001msec)
slat (nsec): min=1563, max=15619, avg=1623.51, stdev=104.72
clat (nsec): min=241, max=77606, avg=4359.35, stdev=1122.87
lat (nsec): min=5650, max=79209, avg=5982.86, stdev=1128.83
clat percentiles (nsec):
| 1.00th=[ 4128], 5.00th=[ 4128], 10.00th=[ 4128], 20.00th=[ 4128],
| 30.00th=[ 4128], 40.00th=[ 4192], 50.00th=[ 4192], 60.00th=[ 4256],
| 70.00th=[ 4256], 80.00th=[ 4320], 90.00th=[ 4384], 95.00th=[ 4448],
| 99.00th=[10048], 99.50th=[10688], 99.90th=[15424], 99.95th=[15552],
| 99.99th=[57088]
bw ( KiB/s): min=80316, max=80630, per=100.00%, avg=80486.62, stdev=102.52, samples=13
iops : min=160632, max=161260, avg=160973.23, stdev=205.05, samples=13
lat (nsec) : 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=98.97%, 20=1.01%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=10.33%, sys=89.64%, ctx=21, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1126460,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=78.6MiB/s (82.4MB/s), 78.6MiB/s-78.6MiB/s (82.4MB/s-82.4MB/s), io=550MiB (577MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751198,
"timestamp_ms" : 1666751198179,
"time" : "Wed Oct 26 04:26:38 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 278675456,
"io_kbytes" : 272144,
"bw_bytes" : 39805092,
"bw" : 38872,
"iops" : 77744.179403,
"runtime" : 7001,
"total_ios" : 544287,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2585,
"max" : 41207,
"mean" : 2875.072484,
"stddev" : 249.042554,
"N" : 544288
},
"clat_ns" : {
"min" : 301,
"max" : 155822,
"mean" : 9423.841889,
"stddev" : 1236.552305,
"N" : 544287,
"percentile" : {
"1.000000" : 9024,
"5.000000" : 9024,
"10.000000" : 9152,
"20.000000" : 9152,
"30.000000" : 9280,
"40.000000" : 9280,
"50.000000" : 9280,
"60.000000" : 9280,
"70.000000" : 9280,
"80.000000" : 9408,
"90.000000" : 9408,
"95.000000" : 9664,
"99.000000" : 15168,
"99.500000" : 15808,
"99.900000" : 19840,
"99.950000" : 26752,
"99.990000" : 55552
}
},
"lat_ns" : {
"min" : 9348,
"max" : 158678,
"mean" : 12298.914446,
"stddev" : 1258.957845,
"N" : 544287
},
"bw_min" : 38691,
"bw_max" : 39063,
"bw_agg" : 100.000000,
"bw_mean" : 38882.357143,
"bw_dev" : 110.504860,
"bw_samples" : 14,
"iops_min" : 77382,
"iops_max" : 78126,
"iops_mean" : 77764.714286,
"iops_stddev" : 221.009721,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.557143,
"sys_cpu" : 47.928571,
"ctx" : 544144,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.011942,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.766412,
"20" : 3.138785,
"50" : 0.065958,
"100" : 0.016535,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=139470: Wed Oct 26 04:26:38 2022
read: IOPS=77.7k, BW=38.0MiB/s (39.8MB/s)(266MiB/7001msec)
slat (nsec): min=2585, max=41207, avg=2875.07, stdev=249.04
clat (nsec): min=301, max=155822, avg=9423.84, stdev=1236.55
lat (usec): min=9, max=158, avg=12.30, stdev= 1.26
clat percentiles (nsec):
| 1.00th=[ 9024], 5.00th=[ 9024], 10.00th=[ 9152], 20.00th=[ 9152],
| 30.00th=[ 9280], 40.00th=[ 9280], 50.00th=[ 9280], 60.00th=[ 9280],
| 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[ 9664],
| 99.00th=[15168], 99.50th=[15808], 99.90th=[19840], 99.95th=[26752],
| 99.99th=[55552]
bw ( KiB/s): min=38691, max=39063, per=100.00%, avg=38882.36, stdev=110.50, samples=14
iops : min=77382, max=78126, avg=77764.71, stdev=221.01, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 10=96.77%, 20=3.14%, 50=0.07%, 100=0.02%, 250=0.01%
cpu : usr=13.56%, sys=47.93%, ctx=544144, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=544287,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=38.0MiB/s (39.8MB/s), 38.0MiB/s-38.0MiB/s (39.8MB/s-39.8MB/s), io=266MiB (279MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751211,
"timestamp_ms" : 1666751211808,
"time" : "Wed Oct 26 04:26:51 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 277933056,
"io_kbytes" : 271419,
"bw_bytes" : 39699050,
"bw" : 38768,
"iops" : 77537.208970,
"runtime" : 7001,
"total_ios" : 542838,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2625,
"max" : 47760,
"mean" : 2836.992208,
"stddev" : 284.296938,
"N" : 542838
},
"clat_ns" : {
"min" : 311,
"max" : 110006,
"mean" : 9445.447319,
"stddev" : 1220.518444,
"N" : 542838,
"percentile" : {
"1.000000" : 9024,
"5.000000" : 9152,
"10.000000" : 9152,
"20.000000" : 9152,
"30.000000" : 9280,
"40.000000" : 9280,
"50.000000" : 9280,
"60.000000" : 9280,
"70.000000" : 9408,
"80.000000" : 9408,
"90.000000" : 9408,
"95.000000" : 9664,
"99.000000" : 15168,
"99.500000" : 15808,
"99.900000" : 20096,
"99.950000" : 27264,
"99.990000" : 56064
}
},
"lat_ns" : {
"min" : 9437,
"max" : 112812,
"mean" : 12282.439527,
"stddev" : 1251.244279,
"N" : 542838
},
"bw_min" : 38539,
"bw_max" : 38959,
"bw_agg" : 100.000000,
"bw_mean" : 38780.285714,
"bw_dev" : 105.847807,
"bw_samples" : 14,
"iops_min" : 77078,
"iops_max" : 77918,
"iops_mean" : 77560.642857,
"iops_stddev" : 211.751526,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.314286,
"sys_cpu" : 48.014286,
"ctx" : 542695,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010685,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.858547,
"20" : 3.030923,
"50" : 0.083819,
"100" : 0.015658,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=139510: Wed Oct 26 04:26:51 2022
read: IOPS=77.5k, BW=37.9MiB/s (39.7MB/s)(265MiB/7001msec)
slat (nsec): min=2625, max=47760, avg=2836.99, stdev=284.30
clat (nsec): min=311, max=110006, avg=9445.45, stdev=1220.52
lat (usec): min=9, max=112, avg=12.28, stdev= 1.25
clat percentiles (nsec):
| 1.00th=[ 9024], 5.00th=[ 9152], 10.00th=[ 9152], 20.00th=[ 9152],
| 30.00th=[ 9280], 40.00th=[ 9280], 50.00th=[ 9280], 60.00th=[ 9280],
| 70.00th=[ 9408], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[ 9664],
| 99.00th=[15168], 99.50th=[15808], 99.90th=[20096], 99.95th=[27264],
| 99.99th=[56064]
bw ( KiB/s): min=38539, max=38959, per=100.00%, avg=38780.29, stdev=105.85, samples=14
iops : min=77078, max=77918, avg=77560.64, stdev=211.75, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 10=96.86%, 20=3.03%, 50=0.08%, 100=0.02%, 250=0.01%
cpu : usr=13.31%, sys=48.01%, ctx=542695, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=542838,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=37.9MiB/s (39.7MB/s), 37.9MiB/s-37.9MiB/s (39.7MB/s-39.7MB/s), io=265MiB (278MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_io_uring-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751225,
"timestamp_ms" : 1666751225436,
"time" : "Wed Oct 26 04:27:05 2022",
"jobs" : [
{
"jobname" : "io_uring-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "io_uring-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "io_uring",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 275073536,
"io_kbytes" : 268626,
"bw_bytes" : 39290606,
"bw" : 38369,
"iops" : 76739.465791,
"runtime" : 7001,
"total_ios" : 537253,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2655,
"max" : 33883,
"mean" : 2914.394468,
"stddev" : 191.365400,
"N" : 537253
},
"clat_ns" : {
"min" : 301,
"max" : 98825,
"mean" : 9502.469501,
"stddev" : 1206.508405,
"N" : 537253,
"percentile" : {
"1.000000" : 9024,
"5.000000" : 9152,
"10.000000" : 9152,
"20.000000" : 9280,
"30.000000" : 9280,
"40.000000" : 9280,
"50.000000" : 9408,
"60.000000" : 9408,
"70.000000" : 9408,
"80.000000" : 9408,
"90.000000" : 9536,
"95.000000" : 9664,
"99.000000" : 15168,
"99.500000" : 15680,
"99.900000" : 19840,
"99.950000" : 25472,
"99.990000" : 57600
}
},
"lat_ns" : {
"min" : 9358,
"max" : 101720,
"mean" : 12416.863969,
"stddev" : 1220.253756,
"N" : 537253
},
"bw_min" : 38246,
"bw_max" : 38615,
"bw_agg" : 100.000000,
"bw_mean" : 38379.428571,
"bw_dev" : 100.427329,
"bw_samples" : 14,
"iops_min" : 76492,
"iops_max" : 77230,
"iops_mean" : 76758.857143,
"iops_stddev" : 200.854657,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.200000,
"sys_cpu" : 46.171429,
"ctx" : 537127,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.011726,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 96.830916,
"20" : 3.088675,
"50" : 0.051372,
"100" : 0.017310,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=139549: Wed Oct 26 04:27:05 2022
read: IOPS=76.7k, BW=37.5MiB/s (39.3MB/s)(262MiB/7001msec)
slat (nsec): min=2655, max=33883, avg=2914.39, stdev=191.37
clat (nsec): min=301, max=98825, avg=9502.47, stdev=1206.51
lat (usec): min=9, max=101, avg=12.42, stdev= 1.22
clat percentiles (nsec):
| 1.00th=[ 9024], 5.00th=[ 9152], 10.00th=[ 9152], 20.00th=[ 9280],
| 30.00th=[ 9280], 40.00th=[ 9280], 50.00th=[ 9408], 60.00th=[ 9408],
| 70.00th=[ 9408], 80.00th=[ 9408], 90.00th=[ 9536], 95.00th=[ 9664],
| 99.00th=[15168], 99.50th=[15680], 99.90th=[19840], 99.95th=[25472],
| 99.99th=[57600]
bw ( KiB/s): min=38246, max=38615, per=100.00%, avg=38379.43, stdev=100.43, samples=14
iops : min=76492, max=77230, avg=76758.86, stdev=200.85, samples=14
lat (nsec) : 500=0.01%
lat (usec) : 10=96.83%, 20=3.09%, 50=0.05%, 100=0.02%
cpu : usr=15.20%, sys=46.17%, ctx=537127, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=537253,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=37.5MiB/s (39.3MB/s), 37.5MiB/s-37.5MiB/s (39.3MB/s-39.3MB/s), io=262MiB (275MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-reference_0.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751239,
"timestamp_ms" : 1666751239082,
"time" : "Wed Oct 26 04:27:19 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 285667328,
"io_kbytes" : 278972,
"bw_bytes" : 40803789,
"bw" : 39847,
"iops" : 79694.900728,
"runtime" : 7001,
"total_ios" : 557944,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2726,
"max" : 35376,
"mean" : 2949.894054,
"stddev" : 248.175186,
"N" : 557944
},
"clat_ns" : {
"min" : 801,
"max" : 72345,
"mean" : 8997.065191,
"stddev" : 1285.566522,
"N" : 557944,
"percentile" : {
"1.000000" : 8640,
"5.000000" : 8640,
"10.000000" : 8640,
"20.000000" : 8768,
"30.000000" : 8768,
"40.000000" : 8768,
"50.000000" : 8768,
"60.000000" : 8896,
"70.000000" : 8896,
"80.000000" : 8896,
"90.000000" : 9024,
"95.000000" : 9152,
"99.000000" : 15168,
"99.500000" : 15808,
"99.900000" : 19328,
"99.950000" : 26240,
"99.990000" : 55552
}
},
"lat_ns" : {
"min" : 9508,
"max" : 75291,
"mean" : 11946.959245,
"stddev" : 1309.323831,
"N" : 557944
},
"bw_min" : 39689,
"bw_max" : 39936,
"bw_agg" : 100.000000,
"bw_mean" : 39863.642857,
"bw_dev" : 67.919191,
"bw_samples" : 14,
"iops_min" : 79378,
"iops_max" : 79872,
"iops_mean" : 79727.357143,
"iops_stddev" : 135.822060,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 14.842857,
"sys_cpu" : 45.585714,
"ctx" : 557800,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 97.375543,
"20" : 2.532512,
"50" : 0.062013,
"100" : 0.019536,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 799685,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5026,
"write_ticks" : 0,
"in_queue" : 5026,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=139589: Wed Oct 26 04:27:19 2022
read: IOPS=79.7k, BW=38.9MiB/s (40.8MB/s)(272MiB/7001msec)
slat (nsec): min=2726, max=35376, avg=2949.89, stdev=248.18
clat (nsec): min=801, max=72345, avg=8997.07, stdev=1285.57
lat (nsec): min=9508, max=75291, avg=11946.96, stdev=1309.32
clat percentiles (nsec):
| 1.00th=[ 8640], 5.00th=[ 8640], 10.00th=[ 8640], 20.00th=[ 8768],
| 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8896],
| 70.00th=[ 8896], 80.00th=[ 8896], 90.00th=[ 9024], 95.00th=[ 9152],
| 99.00th=[15168], 99.50th=[15808], 99.90th=[19328], 99.95th=[26240],
| 99.99th=[55552]
bw ( KiB/s): min=39689, max=39936, per=100.00%, avg=39863.64, stdev=67.92, samples=14
iops : min=79378, max=79872, avg=79727.36, stdev=135.82, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=97.38%, 20=2.53%, 50=0.06%, 100=0.02%
cpu : usr=14.84%, sys=45.59%, ctx=557800, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=557944,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=38.9MiB/s (40.8MB/s), 38.9MiB/s-38.9MiB/s (40.8MB/s-40.8MB/s), io=272MiB (286MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=799685/0, merge=0/0, ticks=5026/0, in_queue=5026, util=99.09%
$ rm /tmp/fio-output_512_1_libaio-reference_1.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751252,
"timestamp_ms" : 1666751252703,
"time" : "Wed Oct 26 04:27:32 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 435968000,
"io_kbytes" : 425750,
"bw_bytes" : 62272246,
"bw" : 60812,
"iops" : 121625.482074,
"runtime" : 7001,
"total_ios" : 851500,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1543,
"max" : 14407,
"mean" : 1658.127843,
"stddev" : 52.610402,
"N" : 851500
},
"clat_ns" : {
"min" : 410,
"max" : 96491,
"mean" : 6259.916447,
"stddev" : 1052.788853,
"N" : 851500,
"percentile" : {
"1.000000" : 5920,
"5.000000" : 5984,
"10.000000" : 5984,
"20.000000" : 6048,
"30.000000" : 6048,
"40.000000" : 6112,
"50.000000" : 6112,
"60.000000" : 6176,
"70.000000" : 6176,
"80.000000" : 6240,
"90.000000" : 6368,
"95.000000" : 6432,
"99.000000" : 11968,
"99.500000" : 12480,
"99.900000" : 15552,
"99.950000" : 15680,
"99.990000" : 56064
}
},
"lat_ns" : {
"min" : 6091,
"max" : 98144,
"mean" : 7918.044290,
"stddev" : 1054.920677,
"N" : 851500
},
"bw_min" : 60701,
"bw_max" : 61033,
"bw_agg" : 100.000000,
"bw_mean" : 60838.642857,
"bw_dev" : 110.521566,
"bw_samples" : 14,
"iops_min" : 121402,
"iops_max" : 122066,
"iops_mean" : 121677.428571,
"iops_stddev" : 221.098208,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 13.700000,
"sys_cpu" : 36.571429,
"ctx" : 851445,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.010000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 98.463065,
"20" : 1.515326,
"50" : 0.010000,
"100" : 0.011861,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 1221975,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5983,
"write_ticks" : 0,
"in_queue" : 5983,
"util" : 99.088748
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=139627: Wed Oct 26 04:27:32 2022
read: IOPS=122k, BW=59.4MiB/s (62.3MB/s)(416MiB/7001msec)
slat (nsec): min=1543, max=14407, avg=1658.13, stdev=52.61
clat (nsec): min=410, max=96491, avg=6259.92, stdev=1052.79
lat (nsec): min=6091, max=98144, avg=7918.04, stdev=1054.92
clat percentiles (nsec):
| 1.00th=[ 5920], 5.00th=[ 5984], 10.00th=[ 5984], 20.00th=[ 6048],
| 30.00th=[ 6048], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[ 6176],
| 70.00th=[ 6176], 80.00th=[ 6240], 90.00th=[ 6368], 95.00th=[ 6432],
| 99.00th=[11968], 99.50th=[12480], 99.90th=[15552], 99.95th=[15680],
| 99.99th=[56064]
bw ( KiB/s): min=60701, max=61033, per=100.00%, avg=60838.64, stdev=110.52, samples=14
iops : min=121402, max=122066, avg=121677.43, stdev=221.10, samples=14
lat (nsec) : 500=0.01%, 750=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=98.46%, 20=1.52%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=13.70%, sys=36.57%, ctx=851445, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=851500,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=59.4MiB/s (62.3MB/s), 59.4MiB/s-59.4MiB/s (62.3MB/s-62.3MB/s), io=416MiB (436MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=1221975/0, merge=0/0, ticks=5983/0, in_queue=5983, util=99.09%
$ rm /tmp/fio-output_512_1_libaio-reference_2.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751266,
"timestamp_ms" : 1666751266367,
"time" : "Wed Oct 26 04:27:46 2022",
"jobs" : [
{
"jobname" : "libaio-reference",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-reference",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "libaio",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 281868800,
"io_kbytes" : 275262,
"bw_bytes" : 40261219,
"bw" : 39317,
"iops" : 78635.194972,
"runtime" : 7001,
"total_ios" : 550525,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2765,
"max" : 34235,
"mean" : 2952.005994,
"stddev" : 196.975536,
"N" : 550525
},
"clat_ns" : {
"min" : 791,
"max" : 76494,
"mean" : 9169.578695,
"stddev" : 1235.513611,
"N" : 550525,
"percentile" : {
"1.000000" : 8768,
"5.000000" : 8768,
"10.000000" : 8896,
"20.000000" : 8896,
"30.000000" : 8896,
"40.000000" : 9024,
"50.000000" : 9024,
"60.000000" : 9024,
"70.000000" : 9024,
"80.000000" : 9152,
"90.000000" : 9152,
"95.000000" : 9280,
"99.000000" : 15040,
"99.500000" : 15680,
"99.900000" : 19328,
"99.950000" : 23936,
"99.990000" : 57600
}
},
"lat_ns" : {
"min" : 9348,
"max" : 79429,
"mean" : 12121.584689,
"stddev" : 1250.717136,
"N" : 550525
},
"bw_min" : 39172,
"bw_max" : 39524,
"bw_agg" : 100.000000,
"bw_mean" : 39335.142857,
"bw_dev" : 102.275538,
"bw_samples" : 14,
"iops_min" : 78344,
"iops_max" : 79048,
"iops_mean" : 78670.428571,
"iops_stddev" : 204.625630,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.157143,
"sys_cpu" : 44.100000,
"ctx" : 550395,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 97.186867,
"20" : 2.744653,
"50" : 0.040507,
"100" : 0.018709,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
],
"disk_util" : [
{
"name" : "nvme0n1",
"read_ios" : 790291,
"write_ios" : 0,
"read_merges" : 0,
"write_merges" : 0,
"read_ticks" : 5056,
"write_ticks" : 0,
"in_queue" : 5056,
"util" : 99.058940
}
]
}
libaio-reference: (groupid=0, jobs=1): err= 0: pid=139665: Wed Oct 26 04:27:46 2022
read: IOPS=78.6k, BW=38.4MiB/s (40.3MB/s)(269MiB/7001msec)
slat (nsec): min=2765, max=34235, avg=2952.01, stdev=196.98
clat (nsec): min=791, max=76494, avg=9169.58, stdev=1235.51
lat (nsec): min=9348, max=79429, avg=12121.58, stdev=1250.72
clat percentiles (nsec):
| 1.00th=[ 8768], 5.00th=[ 8768], 10.00th=[ 8896], 20.00th=[ 8896],
| 30.00th=[ 8896], 40.00th=[ 9024], 50.00th=[ 9024], 60.00th=[ 9024],
| 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9152], 95.00th=[ 9280],
| 99.00th=[15040], 99.50th=[15680], 99.90th=[19328], 99.95th=[23936],
| 99.99th=[57600]
bw ( KiB/s): min=39172, max=39524, per=100.00%, avg=39335.14, stdev=102.28, samples=14
iops : min=78344, max=79048, avg=78670.43, stdev=204.63, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=97.19%, 20=2.74%, 50=0.04%, 100=0.02%
cpu : usr=16.16%, sys=44.10%, ctx=550395, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=550525,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=38.4MiB/s (40.3MB/s), 38.4MiB/s-38.4MiB/s (40.3MB/s-40.3MB/s), io=269MiB (282MB), run=7001-7001msec
Disk stats (read/write):
nvme0n1: ios=790291/0, merge=0/0, ticks=5056/0, in_queue=5056, util=99.06%
$ rm /tmp/fio-output_512_1_libaio-bdev_aio_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751281,
"timestamp_ms" : 1666751281368,
"time" : "Wed Oct 26 04:28:01 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 568576000,
"io_kbytes" : 555250,
"bw_bytes" : 81225142,
"bw" : 79321,
"iops" : 158642.857143,
"runtime" : 7000,
"total_ios" : 1110500,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1613,
"max" : 37350,
"mean" : 1678.867968,
"stddev" : 146.532978,
"N" : 1110500
},
"clat_ns" : {
"min" : 190,
"max" : 115837,
"mean" : 4397.280773,
"stddev" : 1142.935701,
"N" : 1110500,
"percentile" : {
"1.000000" : 4128,
"5.000000" : 4128,
"10.000000" : 4128,
"20.000000" : 4192,
"30.000000" : 4192,
"40.000000" : 4192,
"50.000000" : 4256,
"60.000000" : 4256,
"70.000000" : 4320,
"80.000000" : 4320,
"90.000000" : 4384,
"95.000000" : 4576,
"99.000000" : 9792,
"99.500000" : 11328,
"99.900000" : 15552,
"99.950000" : 15680,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5730,
"max" : 117510,
"mean" : 6076.148740,
"stddev" : 1154.189344,
"N" : 1110500
},
"bw_min" : 79120,
"bw_max" : 79623,
"bw_agg" : 100.000000,
"bw_mean" : 79328.214286,
"bw_dev" : 158.211241,
"bw_samples" : 14,
"iops_min" : 158240,
"iops_max" : 159246,
"iops_mean" : 158656.500000,
"iops_stddev" : 316.477913,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 76.614286,
"sys_cpu" : 23.357143,
"ctx" : 26,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 99.038451,
"20" : 0.941108,
"50" : 0.010000,
"100" : 0.012517,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=139713: Wed Oct 26 04:28:01 2022
read: IOPS=159k, BW=77.5MiB/s (81.2MB/s)(542MiB/7000msec)
slat (nsec): min=1613, max=37350, avg=1678.87, stdev=146.53
clat (nsec): min=190, max=115837, avg=4397.28, stdev=1142.94
lat (usec): min=5, max=117, avg= 6.08, stdev= 1.15
clat percentiles (nsec):
| 1.00th=[ 4128], 5.00th=[ 4128], 10.00th=[ 4128], 20.00th=[ 4192],
| 30.00th=[ 4192], 40.00th=[ 4192], 50.00th=[ 4256], 60.00th=[ 4256],
| 70.00th=[ 4320], 80.00th=[ 4320], 90.00th=[ 4384], 95.00th=[ 4576],
| 99.00th=[ 9792], 99.50th=[11328], 99.90th=[15552], 99.95th=[15680],
| 99.99th=[57088]
bw ( KiB/s): min=79120, max=79623, per=100.00%, avg=79328.21, stdev=158.21, samples=14
iops : min=158240, max=159246, avg=158656.50, stdev=316.48, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=99.04%, 20=0.94%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=76.61%, sys=23.36%, ctx=26, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1110500,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=77.5MiB/s (81.2MB/s), 77.5MiB/s-77.5MiB/s (81.2MB/s-81.2MB/s), io=542MiB (569MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_libaio-bdev_aio_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751296,
"timestamp_ms" : 1666751296479,
"time" : "Wed Oct 26 04:28:16 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 569852416,
"io_kbytes" : 556496,
"bw_bytes" : 81407488,
"bw" : 79499,
"iops" : 158999.000000,
"runtime" : 7000,
"total_ios" : 1112993,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1623,
"max" : 25918,
"mean" : 1684.411642,
"stddev" : 128.915050,
"N" : 1112993
},
"clat_ns" : {
"min" : 210,
"max" : 76463,
"mean" : 4380.684294,
"stddev" : 1145.616557,
"N" : 1112993,
"percentile" : {
"1.000000" : 4128,
"5.000000" : 4128,
"10.000000" : 4128,
"20.000000" : 4128,
"30.000000" : 4192,
"40.000000" : 4192,
"50.000000" : 4256,
"60.000000" : 4256,
"70.000000" : 4256,
"80.000000" : 4320,
"90.000000" : 4384,
"95.000000" : 4512,
"99.000000" : 9792,
"99.500000" : 11200,
"99.900000" : 15424,
"99.950000" : 15680,
"99.990000" : 57600
}
},
"lat_ns" : {
"min" : 5731,
"max" : 78166,
"mean" : 6065.095936,
"stddev" : 1154.036926,
"N" : 1112993
},
"bw_min" : 79322,
"bw_max" : 79715,
"bw_agg" : 100.000000,
"bw_mean" : 79509.214286,
"bw_dev" : 119.767196,
"bw_samples" : 14,
"iops_min" : 158644,
"iops_max" : 159430,
"iops_mean" : 159018.571429,
"iops_stddev" : 239.596364,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 76.442857,
"sys_cpu" : 23.528571,
"ctx" : 26,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 99.035214,
"20" : 0.943762,
"50" : 0.010000,
"100" : 0.013208,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=139762: Wed Oct 26 04:28:16 2022
read: IOPS=159k, BW=77.6MiB/s (81.4MB/s)(543MiB/7000msec)
slat (nsec): min=1623, max=25918, avg=1684.41, stdev=128.92
clat (nsec): min=210, max=76463, avg=4380.68, stdev=1145.62
lat (nsec): min=5731, max=78166, avg=6065.10, stdev=1154.04
clat percentiles (nsec):
| 1.00th=[ 4128], 5.00th=[ 4128], 10.00th=[ 4128], 20.00th=[ 4128],
| 30.00th=[ 4192], 40.00th=[ 4192], 50.00th=[ 4256], 60.00th=[ 4256],
| 70.00th=[ 4256], 80.00th=[ 4320], 90.00th=[ 4384], 95.00th=[ 4512],
| 99.00th=[ 9792], 99.50th=[11200], 99.90th=[15424], 99.95th=[15680],
| 99.99th=[57600]
bw ( KiB/s): min=79322, max=79715, per=100.00%, avg=79509.21, stdev=119.77, samples=14
iops : min=158644, max=159430, avg=159018.57, stdev=239.60, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=99.04%, 20=0.94%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=76.44%, sys=23.53%, ctx=26, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1112993,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=77.6MiB/s (81.4MB/s), 77.6MiB/s-77.6MiB/s (81.4MB/s-81.4MB/s), io=543MiB (570MB), run=7000-7000msec
$ rm /tmp/fio-output_512_1_libaio-bdev_aio_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751311,
"timestamp_ms" : 1666751311692,
"time" : "Wed Oct 26 04:28:31 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_aio",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_aio",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 570132480,
"io_kbytes" : 556770,
"bw_bytes" : 81435863,
"bw" : 79527,
"iops" : 159054.277960,
"runtime" : 7001,
"total_ios" : 1113539,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1603,
"max" : 60874,
"mean" : 1654.208657,
"stddev" : 140.431430,
"N" : 1113540
},
"clat_ns" : {
"min" : 200,
"max" : 114164,
"mean" : 4398.236214,
"stddev" : 1110.230874,
"N" : 1113539,
"percentile" : {
"1.000000" : 4128,
"5.000000" : 4128,
"10.000000" : 4192,
"20.000000" : 4192,
"30.000000" : 4192,
"40.000000" : 4192,
"50.000000" : 4256,
"60.000000" : 4256,
"70.000000" : 4320,
"80.000000" : 4320,
"90.000000" : 4384,
"95.000000" : 4512,
"99.000000" : 9920,
"99.500000" : 10560,
"99.900000" : 15552,
"99.950000" : 15680,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5730,
"max" : 115827,
"mean" : 6052.444891,
"stddev" : 1120.267316,
"N" : 1113539
},
"bw_min" : 79426,
"bw_max" : 79846,
"bw_agg" : 100.000000,
"bw_mean" : 79546.428571,
"bw_dev" : 125.765175,
"bw_samples" : 14,
"iops_min" : 158852,
"iops_max" : 159693,
"iops_mean" : 159092.785714,
"iops_stddev" : 251.830889,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 76.100000,
"sys_cpu" : 23.871429,
"ctx" : 25,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.010000,
"500" : 0.010000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 99.023653,
"20" : 0.955692,
"50" : 0.010000,
"100" : 0.011674,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=139811: Wed Oct 26 04:28:31 2022
read: IOPS=159k, BW=77.7MiB/s (81.4MB/s)(544MiB/7001msec)
slat (nsec): min=1603, max=60874, avg=1654.21, stdev=140.43
clat (nsec): min=200, max=114164, avg=4398.24, stdev=1110.23
lat (usec): min=5, max=115, avg= 6.05, stdev= 1.12
clat percentiles (nsec):
| 1.00th=[ 4128], 5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4192],
| 30.00th=[ 4192], 40.00th=[ 4192], 50.00th=[ 4256], 60.00th=[ 4256],
| 70.00th=[ 4320], 80.00th=[ 4320], 90.00th=[ 4384], 95.00th=[ 4512],
| 99.00th=[ 9920], 99.50th=[10560], 99.90th=[15552], 99.95th=[15680],
| 99.99th=[57088]
bw ( KiB/s): min=79426, max=79846, per=100.00%, avg=79546.43, stdev=125.77, samples=14
iops : min=158852, max=159693, avg=159092.79, stdev=251.83, samples=14
lat (nsec) : 250=0.01%, 500=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=99.02%, 20=0.96%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=76.10%, sys=23.87%, ctx=25, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1113539,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=77.7MiB/s (81.4MB/s), 77.7MiB/s-77.7MiB/s (81.4MB/s-81.4MB/s), io=544MiB (570MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751326,
"timestamp_ms" : 1666751326817,
"time" : "Wed Oct 26 04:28:46 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 549307392,
"io_kbytes" : 536433,
"bw_bytes" : 78461275,
"bw" : 76622,
"iops" : 153244.679332,
"runtime" : 7001,
"total_ios" : 1072866,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1623,
"max" : 14217,
"mean" : 1678.038774,
"stddev" : 119.906182,
"N" : 1072866
},
"clat_ns" : {
"min" : 591,
"max" : 118713,
"mean" : 4583.107354,
"stddev" : 1160.723079,
"N" : 1072866,
"percentile" : {
"1.000000" : 4320,
"5.000000" : 4320,
"10.000000" : 4320,
"20.000000" : 4320,
"30.000000" : 4320,
"40.000000" : 4320,
"50.000000" : 4384,
"60.000000" : 4384,
"70.000000" : 4640,
"80.000000" : 4640,
"90.000000" : 4704,
"95.000000" : 4768,
"99.000000" : 9792,
"99.500000" : 10816,
"99.900000" : 15552,
"99.950000" : 15680,
"99.990000" : 59136
}
},
"lat_ns" : {
"min" : 5881,
"max" : 120406,
"mean" : 6261.146127,
"stddev" : 1168.029557,
"N" : 1072866
},
"bw_min" : 76498,
"bw_max" : 76736,
"bw_agg" : 100.000000,
"bw_mean" : 76644.785714,
"bw_dev" : 66.817983,
"bw_samples" : 14,
"iops_min" : 152997,
"iops_max" : 153472,
"iops_mean" : 153289.642857,
"iops_stddev" : 133.467142,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 36.614286,
"sys_cpu" : 63.357143,
"ctx" : 23,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 99.073137,
"20" : 0.903188,
"50" : 0.010000,
"100" : 0.015379,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139861: Wed Oct 26 04:28:46 2022
read: IOPS=153k, BW=74.8MiB/s (78.5MB/s)(524MiB/7001msec)
slat (nsec): min=1623, max=14217, avg=1678.04, stdev=119.91
clat (nsec): min=591, max=118713, avg=4583.11, stdev=1160.72
lat (usec): min=5, max=120, avg= 6.26, stdev= 1.17
clat percentiles (nsec):
| 1.00th=[ 4320], 5.00th=[ 4320], 10.00th=[ 4320], 20.00th=[ 4320],
| 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4384], 60.00th=[ 4384],
| 70.00th=[ 4640], 80.00th=[ 4640], 90.00th=[ 4704], 95.00th=[ 4768],
| 99.00th=[ 9792], 99.50th=[10816], 99.90th=[15552], 99.95th=[15680],
| 99.99th=[59136]
bw ( KiB/s): min=76498, max=76736, per=100.00%, avg=76644.79, stdev=66.82, samples=14
iops : min=152997, max=153472, avg=153289.64, stdev=133.47, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=99.07%, 20=0.90%, 50=0.01%
lat (usec) : 100=0.02%, 250=0.01%
cpu : usr=36.61%, sys=63.36%, ctx=23, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1072866,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=74.8MiB/s (78.5MB/s), 74.8MiB/s-74.8MiB/s (78.5MB/s-78.5MB/s), io=524MiB (549MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751341,
"timestamp_ms" : 1666751341933,
"time" : "Wed Oct 26 04:29:01 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 548251136,
"io_kbytes" : 535401,
"bw_bytes" : 78310403,
"bw" : 76475,
"iops" : 152950.007142,
"runtime" : 7001,
"total_ios" : 1070803,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1633,
"max" : 14327,
"mean" : 1697.999735,
"stddev" : 113.979161,
"N" : 1070803
},
"clat_ns" : {
"min" : 602,
"max" : 97693,
"mean" : 4606.212420,
"stddev" : 1101.921202,
"N" : 1070803,
"percentile" : {
"1.000000" : 4384,
"5.000000" : 4384,
"10.000000" : 4384,
"20.000000" : 4384,
"30.000000" : 4384,
"40.000000" : 4448,
"50.000000" : 4448,
"60.000000" : 4448,
"70.000000" : 4448,
"80.000000" : 4512,
"90.000000" : 4704,
"95.000000" : 4768,
"99.000000" : 9792,
"99.500000" : 10816,
"99.900000" : 15552,
"99.950000" : 15680,
"99.990000" : 57088
}
},
"lat_ns" : {
"min" : 5921,
"max" : 99376,
"mean" : 6304.212155,
"stddev" : 1109.513640,
"N" : 1070803
},
"bw_min" : 76350,
"bw_max" : 76686,
"bw_agg" : 100.000000,
"bw_mean" : 76496.357143,
"bw_dev" : 105.615855,
"bw_samples" : 14,
"iops_min" : 152700,
"iops_max" : 153372,
"iops_mean" : 152992.928571,
"iops_stddev" : 211.019673,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 29.985714,
"sys_cpu" : 69.985714,
"ctx" : 20,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.000000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 99.040907,
"20" : 0.937054,
"50" : 0.010000,
"100" : 0.013168,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139913: Wed Oct 26 04:29:01 2022
read: IOPS=153k, BW=74.7MiB/s (78.3MB/s)(523MiB/7001msec)
slat (nsec): min=1633, max=14327, avg=1698.00, stdev=113.98
clat (nsec): min=602, max=97693, avg=4606.21, stdev=1101.92
lat (nsec): min=5921, max=99376, avg=6304.21, stdev=1109.51
clat percentiles (nsec):
| 1.00th=[ 4384], 5.00th=[ 4384], 10.00th=[ 4384], 20.00th=[ 4384],
| 30.00th=[ 4384], 40.00th=[ 4448], 50.00th=[ 4448], 60.00th=[ 4448],
| 70.00th=[ 4448], 80.00th=[ 4512], 90.00th=[ 4704], 95.00th=[ 4768],
| 99.00th=[ 9792], 99.50th=[10816], 99.90th=[15552], 99.95th=[15680],
| 99.99th=[57088]
bw ( KiB/s): min=76350, max=76686, per=100.00%, avg=76496.36, stdev=105.62, samples=14
iops : min=152700, max=153372, avg=152992.93, stdev=211.02, samples=14
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=99.04%, 20=0.94%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=29.99%, sys=69.99%, ctx=20, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1070803,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=74.7MiB/s (78.3MB/s), 74.7MiB/s-74.7MiB/s (78.3MB/s-78.3MB/s), io=523MiB (548MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created
$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751357,
"timestamp_ms" : 1666751357029,
"time" : "Wed Oct 26 04:29:17 2022",
"jobs" : [
{
"jobname" : "libaio-bdev_xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-bdev_xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "spdk_bdev",
"spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
"filename" : "Nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 552497664,
"io_kbytes" : 539548,
"bw_bytes" : 78916963,
"bw" : 77067,
"iops" : 154134.695044,
"runtime" : 7001,
"total_ios" : 1079097,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 1613,
"max" : 25488,
"mean" : 1684.209908,
"stddev" : 120.441497,
"N" : 1079097
},
"clat_ns" : {
"min" : 592,
"max" : 76343,
"mean" : 4568.302466,
"stddev" : 1090.399727,
"N" : 1079097,
"percentile" : {
"1.000000" : 4320,
"5.000000" : 4320,
"10.000000" : 4320,
"20.000000" : 4384,
"30.000000" : 4384,
"40.000000" : 4384,
"50.000000" : 4384,
"60.000000" : 4448,
"70.000000" : 4448,
"80.000000" : 4448,
"90.000000" : 4704,
"95.000000" : 4768,
"99.000000" : 9792,
"99.500000" : 11328,
"99.900000" : 15552,
"99.950000" : 15680,
"99.990000" : 56576
}
},
"lat_ns" : {
"min" : 5861,
"max" : 78016,
"mean" : 6252.512374,
"stddev" : 1098.558092,
"N" : 1079097
},
"bw_min" : 76780,
"bw_max" : 77307,
"bw_agg" : 100.000000,
"bw_mean" : 77088.357143,
"bw_dev" : 198.919236,
"bw_samples" : 14,
"iops_min" : 153560,
"iops_max" : 154614,
"iops_mean" : 154176.714286,
"iops_stddev" : 397.838473,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 29.242857,
"sys_cpu" : 70.728571,
"ctx" : 22,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.010000,
"10" : 99.054580,
"20" : 0.927349,
"50" : 0.010000,
"100" : 0.010842,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139972: Wed Oct 26 04:29:17 2022
read: IOPS=154k, BW=75.3MiB/s (78.9MB/s)(527MiB/7001msec)
slat (nsec): min=1613, max=25488, avg=1684.21, stdev=120.44
clat (nsec): min=592, max=76343, avg=4568.30, stdev=1090.40
lat (nsec): min=5861, max=78016, avg=6252.51, stdev=1098.56
clat percentiles (nsec):
| 1.00th=[ 4320], 5.00th=[ 4320], 10.00th=[ 4320], 20.00th=[ 4384],
| 30.00th=[ 4384], 40.00th=[ 4384], 50.00th=[ 4384], 60.00th=[ 4448],
| 70.00th=[ 4448], 80.00th=[ 4448], 90.00th=[ 4704], 95.00th=[ 4768],
| 99.00th=[ 9792], 99.50th=[11328], 99.90th=[15552], 99.95th=[15680],
| 99.99th=[56576]
bw ( KiB/s): min=76780, max=77307, per=100.00%, avg=77088.36, stdev=198.92, samples=14
iops : min=153560, max=154614, avg=154176.71, stdev=397.84, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=99.05%, 20=0.93%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=29.24%, sys=70.73%, ctx=22, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1079097,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=75.3MiB/s (78.9MB/s), 75.3MiB/s-75.3MiB/s (78.9MB/s-78.9MB/s), io=527MiB (552MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751370,
"timestamp_ms" : 1666751370821,
"time" : "Wed Oct 26 04:29:30 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 263074816,
"io_kbytes" : 256909,
"bw_bytes" : 37576748,
"bw" : 36696,
"iops" : 73392.086845,
"runtime" : 7001,
"total_ios" : 513818,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2856,
"max" : 24596,
"mean" : 3174.835060,
"stddev" : 192.695012,
"N" : 513818
},
"clat_ns" : {
"min" : 942,
"max" : 70743,
"mean" : 9889.292977,
"stddev" : 1172.701180,
"N" : 513818,
"percentile" : {
"1.000000" : 8512,
"5.000000" : 9536,
"10.000000" : 9664,
"20.000000" : 9664,
"30.000000" : 9792,
"40.000000" : 9792,
"50.000000" : 9792,
"60.000000" : 9792,
"70.000000" : 9792,
"80.000000" : 9792,
"90.000000" : 9920,
"95.000000" : 10048,
"99.000000" : 15040,
"99.500000" : 16192,
"99.900000" : 20096,
"99.950000" : 24960,
"99.990000" : 58112
}
},
"lat_ns" : {
"min" : 8586,
"max" : 73798,
"mean" : 13064.128038,
"stddev" : 1183.145219,
"N" : 513818
},
"bw_min" : 36599,
"bw_max" : 36839,
"bw_agg" : 100.000000,
"bw_mean" : 36707.785714,
"bw_dev" : 78.708985,
"bw_samples" : 14,
"iops_min" : 73198,
"iops_max" : 73678,
"iops_mean" : 73415.428571,
"iops_stddev" : 157.295271,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.328571,
"sys_cpu" : 47.942857,
"ctx" : 513642,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010120,
"4" : 0.000000,
"10" : 94.480731,
"20" : 5.399772,
"50" : 0.087969,
"100" : 0.019657,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=140012: Wed Oct 26 04:29:30 2022
read: IOPS=73.4k, BW=35.8MiB/s (37.6MB/s)(251MiB/7001msec)
slat (nsec): min=2856, max=24596, avg=3174.84, stdev=192.70
clat (nsec): min=942, max=70743, avg=9889.29, stdev=1172.70
lat (nsec): min=8586, max=73798, avg=13064.13, stdev=1183.15
clat percentiles (nsec):
| 1.00th=[ 8512], 5.00th=[ 9536], 10.00th=[ 9664], 20.00th=[ 9664],
| 30.00th=[ 9792], 40.00th=[ 9792], 50.00th=[ 9792], 60.00th=[ 9792],
| 70.00th=[ 9792], 80.00th=[ 9792], 90.00th=[ 9920], 95.00th=[10048],
| 99.00th=[15040], 99.50th=[16192], 99.90th=[20096], 99.95th=[24960],
| 99.99th=[58112]
bw ( KiB/s): min=36599, max=36839, per=100.00%, avg=36707.79, stdev=78.71, samples=14
iops : min=73198, max=73678, avg=73415.43, stdev=157.30, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=94.48%, 20=5.40%, 50=0.09%, 100=0.02%
cpu : usr=15.33%, sys=47.94%, ctx=513642, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=513818,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=35.8MiB/s (37.6MB/s), 35.8MiB/s-35.8MiB/s (37.6MB/s-37.6MB/s), io=251MiB (263MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751384,
"timestamp_ms" : 1666751384484,
"time" : "Wed Oct 26 04:29:44 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 345976320,
"io_kbytes" : 337867,
"bw_bytes" : 49418128,
"bw" : 48259,
"iops" : 96519.782888,
"runtime" : 7001,
"total_ios" : 675735,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2174,
"max" : 24436,
"mean" : 2431.272401,
"stddev" : 151.964015,
"N" : 675735
},
"clat_ns" : {
"min" : 701,
"max" : 75923,
"mean" : 7477.687172,
"stddev" : 1131.118569,
"N" : 675735,
"percentile" : {
"1.000000" : 7136,
"5.000000" : 7200,
"10.000000" : 7200,
"20.000000" : 7264,
"30.000000" : 7264,
"40.000000" : 7264,
"50.000000" : 7328,
"60.000000" : 7328,
"70.000000" : 7392,
"80.000000" : 7392,
"90.000000" : 7584,
"95.000000" : 7712,
"99.000000" : 14016,
"99.500000" : 15040,
"99.900000" : 16064,
"99.950000" : 17024,
"99.990000" : 56064
}
},
"lat_ns" : {
"min" : 7504,
"max" : 78317,
"mean" : 9908.959573,
"stddev" : 1141.464837,
"N" : 675735
},
"bw_min" : 48137,
"bw_max" : 48485,
"bw_agg" : 100.000000,
"bw_mean" : 48272.857143,
"bw_dev" : 96.448358,
"bw_samples" : 14,
"iops_min" : 96274,
"iops_max" : 96970,
"iops_mean" : 96545.785714,
"iops_stddev" : 192.890634,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 16.128571,
"sys_cpu" : 46.542857,
"ctx" : 675583,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.010000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.000000,
"4" : 0.010000,
"10" : 98.266776,
"20" : 1.699483,
"50" : 0.011839,
"100" : 0.013023,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=140052: Wed Oct 26 04:29:44 2022
read: IOPS=96.5k, BW=47.1MiB/s (49.4MB/s)(330MiB/7001msec)
slat (nsec): min=2174, max=24436, avg=2431.27, stdev=151.96
clat (nsec): min=701, max=75923, avg=7477.69, stdev=1131.12
lat (nsec): min=7504, max=78317, avg=9908.96, stdev=1141.46
clat percentiles (nsec):
| 1.00th=[ 7136], 5.00th=[ 7200], 10.00th=[ 7200], 20.00th=[ 7264],
| 30.00th=[ 7264], 40.00th=[ 7264], 50.00th=[ 7328], 60.00th=[ 7328],
| 70.00th=[ 7392], 80.00th=[ 7392], 90.00th=[ 7584], 95.00th=[ 7712],
| 99.00th=[14016], 99.50th=[15040], 99.90th=[16064], 99.95th=[17024],
| 99.99th=[56064]
bw ( KiB/s): min=48137, max=48485, per=100.00%, avg=48272.86, stdev=96.45, samples=14
iops : min=96274, max=96970, avg=96545.79, stdev=192.89, samples=14
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 4=0.01%, 10=98.27%, 20=1.70%, 50=0.01%, 100=0.01%
cpu : usr=16.13%, sys=46.54%, ctx=675583, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=675735,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=47.1MiB/s (49.4MB/s), 47.1MiB/s-47.1MiB/s (49.4MB/s-49.4MB/s), io=330MiB (346MB), run=7001-7001msec
$ rm /tmp/fio-output_512_1_libaio-xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
"fio version" : "fio-3.32-dirty",
"timestamp" : 1666751398,
"timestamp_ms" : 1666751398140,
"time" : "Wed Oct 26 04:29:58 2022",
"jobs" : [
{
"jobname" : "libaio-xnvme",
"groupid" : 0,
"error" : 0,
"eta" : 0,
"elapsed" : 11,
"job options" : {
"name" : "libaio-xnvme",
"rw" : "randread",
"size" : "12G",
"iodepth" : "1",
"bs" : "512",
"direct" : "1",
"thread" : "1",
"time_based" : "1",
"runtime" : "7",
"ramp_time" : "3",
"norandommap" : "1",
"allow_file_create" : "0",
"ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
"xnvme_be" : "linux",
"xnvme_async" : "libaio",
"xnvme_sync" : "nvme",
"xnvme_admin" : "nvme",
"xnvme_dev_nsid" : "1",
"filename" : "/dev/nvme0n1",
"cpus_allowed" : "1"
},
"read" : {
"io_bytes" : 264449536,
"io_kbytes" : 258251,
"bw_bytes" : 37773108,
"bw" : 36887,
"iops" : 73775.460648,
"runtime" : 7001,
"total_ios" : 516502,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 2955,
"max" : 35957,
"mean" : 3190.098706,
"stddev" : 227.777047,
"N" : 516503
},
"clat_ns" : {
"min" : 892,
"max" : 117179,
"mean" : 9770.164466,
"stddev" : 1173.998407,
"N" : 516502,
"percentile" : {
"1.000000" : 8256,
"5.000000" : 9408,
"10.000000" : 9536,
"20.000000" : 9536,
"30.000000" : 9664,
"40.000000" : 9664,
"50.000000" : 9664,
"60.000000" : 9664,
"70.000000" : 9664,
"80.000000" : 9792,
"90.000000" : 9792,
"95.000000" : 9920,
"99.000000" : 14912,
"99.500000" : 16064,
"99.900000" : 20096,
"99.950000" : 25216,
"99.990000" : 55552
}
},
"lat_ns" : {
"min" : 9898,
"max" : 120415,
"mean" : 12960.263335,
"stddev" : 1193.805394,
"N" : 516502
},
"bw_min" : 36783,
"bw_max" : 37012,
"bw_agg" : 100.000000,
"bw_mean" : 36898.714286,
"bw_dev" : 75.942720,
"bw_samples" : 14,
"iops_min" : 73566,
"iops_max" : 74024,
"iops_mean" : 73797.714286,
"iops_stddev" : 151.591774,
"iops_samples" : 14
},
"write" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"trim" : {
"io_bytes" : 0,
"io_kbytes" : 0,
"bw_bytes" : 0,
"bw" : 0,
"iops" : 0.000000,
"runtime" : 0,
"total_ios" : 0,
"short_ios" : 0,
"drop_ios" : 0,
"slat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"clat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
},
"bw_min" : 0,
"bw_max" : 0,
"bw_agg" : 0.000000,
"bw_mean" : 0.000000,
"bw_dev" : 0.000000,
"bw_samples" : 0,
"iops_min" : 0,
"iops_max" : 0,
"iops_mean" : 0.000000,
"iops_stddev" : 0.000000,
"iops_samples" : 0
},
"sync" : {
"total_ios" : 0,
"lat_ns" : {
"min" : 0,
"max" : 0,
"mean" : 0.000000,
"stddev" : 0.000000,
"N" : 0
}
},
"job_runtime" : 7000,
"usr_cpu" : 15.985714,
"sys_cpu" : 47.085714,
"ctx" : 516350,
"majf" : 0,
"minf" : 0,
"iodepth_level" : {
"1" : 100.000000,
"2" : 0.000000,
"4" : 0.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
">=64" : 0.000000
},
"iodepth_submit" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"iodepth_complete" : {
"0" : 0.000000,
"4" : 100.000000,
"8" : 0.000000,
"16" : 0.000000,
"32" : 0.000000,
"64" : 0.000000,
">=64" : 0.000000
},
"latency_ns" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.010000
},
"latency_us" : {
"2" : 0.010000,
"4" : 0.000000,
"10" : 95.703211,
"20" : 4.180429,
"50" : 0.089061,
"100" : 0.017812,
"250" : 0.010000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000
},
"latency_ms" : {
"2" : 0.000000,
"4" : 0.000000,
"10" : 0.000000,
"20" : 0.000000,
"50" : 0.000000,
"100" : 0.000000,
"250" : 0.000000,
"500" : 0.000000,
"750" : 0.000000,
"1000" : 0.000000,
"2000" : 0.000000,
">=2000" : 0.000000
},
"latency_depth" : 1,
"latency_target" : 0,
"latency_percentile" : 100.000000,
"latency_window" : 0
}
]
}
libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=140092: Wed Oct 26 04:29:58 2022
read: IOPS=73.8k, BW=36.0MiB/s (37.8MB/s)(252MiB/7001msec)
slat (nsec): min=2955, max=35957, avg=3190.10, stdev=227.78
clat (nsec): min=892, max=117179, avg=9770.16, stdev=1174.00
lat (usec): min=9, max=120, avg=12.96, stdev= 1.19
clat percentiles (nsec):
| 1.00th=[ 8256], 5.00th=[ 9408], 10.00th=[ 9536], 20.00th=[ 9536],
| 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[ 9664], 60.00th=[ 9664],
| 70.00th=[ 9664], 80.00th=[ 9792], 90.00th=[ 9792], 95.00th=[ 9920],
| 99.00th=[14912], 99.50th=[16064], 99.90th=[20096], 99.95th=[25216],
| 99.99th=[55552]
bw ( KiB/s): min=36783, max=37012, per=100.00%, avg=36898.71, stdev=75.94, samples=14
iops : min=73566, max=74024, avg=73797.71, stdev=151.59, samples=14
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=95.70%, 20=4.18%, 50=0.09%, 100=0.02%
lat (usec) : 250=0.01%
cpu : usr=15.99%, sys=47.09%, ctx=516350, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=516502,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=36.0MiB/s (37.8MB/s), 36.0MiB/s-36.0MiB/s (37.8MB/s-37.8MB/s), io=252MiB (264MB), run=7001-7001msec
Steps uses worklets. A description of the worklet used by
the current step is provided below. The description is extracted from the docstring
of the worklet and provided here in verbatim / as-is.
Run fio...
{'description': '\n Run fio...\n',
'extras': {},
'id': '004_plot_fio',
'name': 'plot_fio',
'nr': 4,
'status': {'elapsed': 0.6777725219726562,
'failed': 0,
'passed': 1,
'skipped': 0,
'started': 0.0},
'uses': 'plot',
'with': {'path': 'cijoe-output',
'plot_attributes': 'plot-attributes-512.yaml',
'tool': 'fio'}}